**1. Introduction**

### *1.1. Introduction to Our Work*

Multimodality, or combining several modes in order to communicate information, is an important technique in the field of HCI (Human Computer Interaction). Since each type of interface and interaction has its own strengths and weaknesses, the combination of many modes of communication results in a more efficient user-machine communication compared to the one accomplished through a system that is based on only one type of user interaction. As a result, researchers have extensively studied different types of interaction and sensing modalities that could be used by combining several of them together in assistive technology solutions for the VIP (Visually Impaired People) [1]. However, even though research related to cross-modal associations based on haptic interfaces exists, research about thermal cues as a possible sensing modality for assistive technologies for VIP is quite small, especially when compared to other haptic interfaces such as vibrotactile actuators.

In this work, the correlation between temperature and depth was studied and evaluated with real users in order to find out a proper mapping between temperature and depth. After that, the algorithm was applied to two different contexts: for conveying depth and for conveying color-based depth (effect known as chromostereopsis [2]). Also, a physical prototype was designed, implemented, and evaluated always in the context of tactile artwork exploration for VIP. The system contains an array of petlier devices in which the artwork can be installed. The user can explore the artwork with the hands while feeling the temperature of the di fferent features of the artwork as a way of conveying the depth levels. All information related to the prototype and the mapping algorithm will be given in the corresponding sections.

This work is the continuation of a series of works whose main goal is the development of a multi-modal tactile artwork system that can help blind people appreciate bi-dimensional pieces of art. The main system, presented in [3,4], consisted of a 3D-printed *2.5D* relief replica of an artwork installed on top of the multimodal guide system. In those works, the *2.5D* relief model is a replica of the bi-dimensional artwork but with a z dimensional depth added to the di fferent features. The user's finger positions were recognized in real time through the use of conductive paint and real-time audio feedback was triggered by double or triple tapping on the di fferent features of the artwork. Double tapping activated explanatory audio feedback and triple tapping activated a sound e ffect feedback related to what is touched. A user voice interface was also added to facilitate the control of the system by giving the users the chance of using their voice as input. Feedback from the visually impaired users during the tests encouraged us to add thermal interaction to the system, first for representing colors, as can be seen in [5]. There, a double petlier device was used to communicate a total of 54 di fferent colors to the visually impaired user through temperature cues. Lastly, exploring the possibilities of thermal interaction in the context of artwork tactile exploration led us to the current work, in which temperature cues were used as a way of conveying depth levels of the di fferent features of the artwork.

To sum up, in this work thermal interaction, a way of mapping it to depth, and the applications of that thermal-depth mapping will be explored with the goal of assigning temperature to objects in the painting that enables conveying depth and/or color-based depth. Tests were performed to a total of 18 sighted users and six visually impaired users both during the mapping algorithm design and after developing a tactile temperature prototype artwork model to assess the potentials of thermal interaction for recognizing depth and color-depth in tactile art appreciation. These tests consisted of both prototype testing and interviews with the users. The tests showed both an existing correlation between depth and temperature and that the mapping based on that correlation successes on conveying depth in a new way during artwork tactile exploration. Also, it proved to be a promising technique for improving visually impaired people's artwork exploration assistive devices. Particularly, eight users assessed the similarity between the depth feeling created by the temperature tactile perception and the one which arose from the visual perception. The results showed that the temperature-depth mapping algorithm was able to successfully translate the visual depth feeling into temperature cues. All these results and tests will be shown in the corresponding section.

### *1.2. Background and Related Work*

### 1.2.1. Thermal Perception and Thermal Interaction

Humans are able to detect temperature and temperature variations thanks to the thermoreceptors located in the dermal and epidermal skin layers. These thermoreceptors located in the skin code the relative changes and the absolute temperature and send the information to the brain. Thermoreceptors can be of two types: high-threshold receptors and low-threshold receptors. Low-threshold receptors are activated by temperatures that fall within the range of 15 ◦C and 45 ◦C. On the contrary, high-threshold detectors are activated by temperatures falling outside of that range. In general, within the 15–45 ◦C range, the activation of low-threshold receptors are not accompanied by pain. On the contrary, any temperature outside of that range can be painful and temperatures below 0 ◦C or above 50 ◦C can even cause tissue damage [6].

The non-painful temperature range has been used extensively for adding thermal feedback to a large variety of applications. For example, in [7] the authors designed structured thermal cues for conveying icons when using the phone. They created both thermal icons and intramodal tactile icons that mixed both thermal cues and vibrotactile cues. The users were successfully able to identify most of the icons correctly. Another interesting application of thermal interaction is the one found in [8]. There, thermal cues are used for car driving assistance. Thermal cues are provided in both sides of the steering wheel indicating the driver which way to turn next. These works are some examples of all the potential possibilities of thermal interactions in the field of HCI. Similarly, our work contemplates possible new applications of thermal interaction but in the context of tactile artwork exploration by VIP.

### 1.2.2. Thermal Interaction for Assistive Devices

There has been some research about thermal interaction in the context of mobile device applications and even some design guidelines about it [9]. However, the research about thermal interactive assistive devices for the VIP is not extensive and only a few examples exist. In [10], a system for VIP to feel a virtual sun while exploring virtual environments is presented. The virtual sun is produced by means of a device consisting of twelve infrared lamps. Also, in [5] a thermal interactive assistive device was used to aid VIP know the color of the di fferent features of an artwork. In spite of the fact that this works also explore thermal interaction in the context of artistic or recreational activities, there are no other common traits with our present work. The present research focuses on the existing correlation between depth and temperature and some of the potential applications which arise from it for aiding visually impaired users explore tactile artworks. Semantic applications of temperature (such as recreating a virtual sun) or temperature-color mapping are not part of this work's scope, which instead focuses on temperature-depth correlation, possible mappings, and derived applications.

### 1.2.3. Thermal Interaction for Artwork Exploration

Thermal interaction has already been used in the context of artwork exploration for the VIP. Nevertheless, the common way of applying it has been using temperature as a way of conveying color, since previous research has suggested the existence of a color-temperature association given by the warm-cool spectrum, based in the amount of red and blue. This led research to investigate about the possible mapping between temperature and color, some of them with an art exploration application in mind, such as in [5,10]. While both of those works focused on the design of a tactile-thermal display for haptic exploration of paintings with temperature conveying color, in the case of [10], this was done only as a concept design, without implementation. On the other hand, in [5] a whole tactile-thermal display was prototyped and user tests, as well as interviews, were performed. The results showed not only that the users were able to recognize colors by feeling temperature, but also that they enjoyed the thermal interaction and that the use of thermal interaction in the context of tactile artworks was promising. Our present work follows the same path and tries to extend the possible uses of thermal interaction for tactile artwork exploration. However, this work does not focus on conveying color but, instead, on conveying the depth of the di fferent features of the artwork.

### 1.2.4. Assistive Devices for Communicating Depth

There are many ways of communicating depth, or how near or far an object is, to the VIP, both in a general context and also in the particular context of artwork tactile exploration. However, to our knowledge, none of them has used temperature interaction as a way of communicating depth. In non-artistic contexts, VIP are usually communicated how far or near objects are when using navigation assistive device. For example, in [11] an obstacle detection system was implemented in a navigation assistive device. The system used ultrasound to detect the nearest obstacle via stereoscopic sonar system. Once the obstacle was detected, a vibrotactile feedback was used to inform the blind person about the obstacle's location.

Similarly, in artistic contexts there are several methods for communicating depth levels to VIP. We can find all of them at museums in what are called 'guides'. There are several types of guides, but all of them share the same goal: giving information about the artwork to the visually impaired user. This information might or might not include depth, but in general it does include it. The three types of guides are: audio guides, relief guides, and volumetric guides. Audio guides are the most common method for providing descriptions of the artwork to VIP. They provide a verbal description of the

di fferent features of the artwork. Most of the time, the relative distance and depth of the di fferent features from the artwork are also explained with words.

Relief guides are a type of tactile guides that allow the user to comprehend the visual information by means of the sense of touch. In the case of relief guides, the artwork is translated into an embossed picture or relieve image. Sometimes, only the contour of the di fferent features is salient, so the user can only feel those contours. In those types of relief guides, the depth information is not given to the user in a tactile way. However, in many instances, all the features of the artwork are extruded and have a third dimension added to them, which allows the user to feel the depth of the di fferent artwork features with the fingers. This type of relief guides can be called *2.5D* guide. An example of a *2.5D* image artwork model can be seen in [12].

The other type of tactile guides are volumetric guides. These are completely three-dimensional volumetric works that the visually impaired user is able to explore by touching. They are usually the types of guides that gives more information through tactile interaction. However, if the original work is bi-dimensional, it is usually hard or not possible to transform it into a volumetric form. As a result, most of the research related to bi-dimensional art for visually impaired people takes advantage of audio and relief guides, rather than volumetric guides.

In the case of this work, the copy of the artwork is a relief model based on tactile paper embossing technology. The result is a relief guide where only the contours of the features are embossed. This type of thin tactile paper allow us to add thermal interaction so the user can feel the temperature in the fingers while exploring the relief model. This temperature is what communicates depth to the user by way of a novel and intuitive temperature-depth mapping algorithm whose foundation will be explained in the next section.
