Thermal Interaction for Improving Tactile Artwork Depth and Color-Depth Appreciation for Visually Impaired People

Abstract

Visually impaired people can take advantage of multimodal systems in which visual information is communicated through different modes of interaction and types of feedback. Among the possible interaction modes, thermal interaction in the context of assistive devices for visually impaired people lacks research in spite of its potential. In this paper, we propose a temperature-depth mapping algorithm and a thermal display system to convey depth and depth-color of artworks’ features in the context of tactile exploration by visually impaired people. Tests with a total of 18 sighted users and six visually impaired users were performed both during the mapping algorithm design and after developing a tactile temperature prototype artwork model to assess the potentials of thermal interaction for recognizing depth and color-depth in tactile art appreciation. These tests showed both an existing correlation between depth and temperature and that the mapping based on that correlation is appropriate for conveying depth during artwork tactile exploration.

1. Introduction

1.1. Introduction to Our Work

Multimodality, or combining several modes in order to communicate information, is an important technique in the field of HCI (Human Computer Interaction). Since each type of interface and interaction has its own strengths and weaknesses, the combination of many modes of communication results in a more efficient user-machine communication compared to the one accomplished through a system that is based on only one type of user interaction. As a result, researchers have extensively studied different types of interaction and sensing modalities that could be used by combining several of them together in assistive technology solutions for the VIP (Visually Impaired People) [1]. However, even though research related to cross-modal associations based on haptic interfaces exists, research about thermal cues as a possible sensing modality for assistive technologies for VIP is quite small, especially when compared to other haptic interfaces such as vibrotactile actuators.

In this work, the correlation between temperature and depth was studied and evaluated with real users in order to find out a proper mapping between temperature and depth. After that, the algorithm was applied to two different contexts: for conveying depth and for conveying color-based depth (effect known as chromostereopsis [2]). Also, a physical prototype was designed, implemented, and evaluated always in the context of tactile artwork exploration for VIP. The system contains an array of petlier devices in which the artwork can be installed. The user can explore the artwork with the hands while feeling the temperature of the different features of the artwork as a way of conveying the depth levels. All information related to the prototype and the mapping algorithm will be given in the corresponding sections.

This work is the continuation of a series of works whose main goal is the development of a multi-modal tactile artwork system that can help blind people appreciate bi-dimensional pieces of art. The main system, presented in [3,4], consisted of a 3D-printed 2.5D relief replica of an artwork installed on top of the multimodal guide system. In those works, the 2.5D relief model is a replica of the bi-dimensional artwork but with a z dimensional depth added to the different features. The user’s finger positions were recognized in real time through the use of conductive paint and real-time audio feedback was triggered by double or triple tapping on the different features of the artwork. Double tapping activated explanatory audio feedback and triple tapping activated a sound effect feedback related to what is touched. A user voice interface was also added to facilitate the control of the system by giving the users the chance of using their voice as input. Feedback from the visually impaired users during the tests encouraged us to add thermal interaction to the system, first for representing colors, as can be seen in [5]. There, a double petlier device was used to communicate a total of 54 different colors to the visually impaired user through temperature cues. Lastly, exploring the possibilities of thermal interaction in the context of artwork tactile exploration led us to the current work, in which temperature cues were used as a way of conveying depth levels of the different features of the artwork.

To sum up, in this work thermal interaction, a way of mapping it to depth, and the applications of that thermal-depth mapping will be explored with the goal of assigning temperature to objects in the painting that enables conveying depth and/or color-based depth. Tests were performed to a total of 18 sighted users and six visually impaired users both during the mapping algorithm design and after developing a tactile temperature prototype artwork model to assess the potentials of thermal interaction for recognizing depth and color-depth in tactile art appreciation. These tests consisted of both prototype testing and interviews with the users. The tests showed both an existing correlation between depth and temperature and that the mapping based on that correlation successes on conveying depth in a new way during artwork tactile exploration. Also, it proved to be a promising technique for improving visually impaired people’s artwork exploration assistive devices. Particularly, eight users assessed the similarity between the depth feeling created by the temperature tactile perception and the one which arose from the visual perception. The results showed that the temperature-depth mapping algorithm was able to successfully translate the visual depth feeling into temperature cues. All these results and tests will be shown in the corresponding section.

1.2. Background and Related Work

1.2.1. Thermal Perception and Thermal Interaction

Humans are able to detect temperature and temperature variations thanks to the thermoreceptors located in the dermal and epidermal skin layers. These thermoreceptors located in the skin code the relative changes and the absolute temperature and send the information to the brain. Thermoreceptors can be of two types: high-threshold receptors and low-threshold receptors. Low-threshold receptors are activated by temperatures that fall within the range of 15 °C and 45 °C. On the contrary, high-threshold detectors are activated by temperatures falling outside of that range. In general, within the 15–45 °C range, the activation of low-threshold receptors are not accompanied by pain. On the contrary, any temperature outside of that range can be painful and temperatures below 0 °C or above 50 °C can even cause tissue damage [6].

The non-painful temperature range has been used extensively for adding thermal feedback to a large variety of applications. For example, in [7] the authors designed structured thermal cues for conveying icons when using the phone. They created both thermal icons and intramodal tactile icons that mixed both thermal cues and vibrotactile cues. The users were successfully able to identify most of the icons correctly. Another interesting application of thermal interaction is the one found in [8]. There, thermal cues are used for car driving assistance. Thermal cues are provided in both sides of the steering wheel indicating the driver which way to turn next. These works are some examples of all the potential possibilities of thermal interactions in the field of HCI. Similarly, our work contemplates possible new applications of thermal interaction but in the context of tactile artwork exploration by VIP.

1.2.2. Thermal Interaction for Assistive Devices

There has been some research about thermal interaction in the context of mobile device applications and even some design guidelines about it [9]. However, the research about thermal interactive assistive devices for the VIP is not extensive and only a few examples exist. In [10], a system for VIP to feel a virtual sun while exploring virtual environments is presented. The virtual sun is produced by means of a device consisting of twelve infrared lamps. Also, in [5] a thermal interactive assistive device was used to aid VIP know the color of the different features of an artwork. In spite of the fact that this works also explore thermal interaction in the context of artistic or recreational activities, there are no other common traits with our present work. The present research focuses on the existing correlation between depth and temperature and some of the potential applications which arise from it for aiding visually impaired users explore tactile artworks. Semantic applications of temperature (such as recreating a virtual sun) or temperature-color mapping are not part of this work’s scope, which instead focuses on temperature-depth correlation, possible mappings, and derived applications.

1.2.3. Thermal Interaction for Artwork Exploration

Thermal interaction has already been used in the context of artwork exploration for the VIP. Nevertheless, the common way of applying it has been using temperature as a way of conveying color, since previous research has suggested the existence of a color-temperature association given by the warm-cool spectrum, based in the amount of red and blue. This led research to investigate about the possible mapping between temperature and color, some of them with an art exploration application in mind, such as in [5,10]. While both of those works focused on the design of a tactile-thermal display for haptic exploration of paintings with temperature conveying color, in the case of [10], this was done only as a concept design, without implementation. On the other hand, in [5] a whole tactile-thermal display was prototyped and user tests, as well as interviews, were performed. The results showed not only that the users were able to recognize colors by feeling temperature, but also that they enjoyed the thermal interaction and that the use of thermal interaction in the context of tactile artworks was promising. Our present work follows the same path and tries to extend the possible uses of thermal interaction for tactile artwork exploration. However, this work does not focus on conveying color but, instead, on conveying the depth of the different features of the artwork.

1.2.4. Assistive Devices for Communicating Depth

There are many ways of communicating depth, or how near or far an object is, to the VIP, both in a general context and also in the particular context of artwork tactile exploration. However, to our knowledge, none of them has used temperature interaction as a way of communicating depth. In non-artistic contexts, VIP are usually communicated how far or near objects are when using navigation assistive device. For example, in [11] an obstacle detection system was implemented in a navigation assistive device. The system used ultrasound to detect the nearest obstacle via stereoscopic sonar system. Once the obstacle was detected, a vibrotactile feedback was used to inform the blind person about the obstacle’s location.

Similarly, in artistic contexts there are several methods for communicating depth levels to VIP. We can find all of them at museums in what are called ‘guides’. There are several types of guides, but all of them share the same goal: giving information about the artwork to the visually impaired user. This information might or might not include depth, but in general it does include it. The three types of guides are: audio guides, relief guides, and volumetric guides. Audio guides are the most common method for providing descriptions of the artwork to VIP. They provide a verbal description of the different features of the artwork. Most of the time, the relative distance and depth of the different features from the artwork are also explained with words.

Relief guides are a type of tactile guides that allow the user to comprehend the visual information by means of the sense of touch. In the case of relief guides, the artwork is translated into an embossed picture or relieve image. Sometimes, only the contour of the different features is salient, so the user can only feel those contours. In those types of relief guides, the depth information is not given to the user in a tactile way. However, in many instances, all the features of the artwork are extruded and have a third dimension added to them, which allows the user to feel the depth of the different artwork features with the fingers. This type of relief guides can be called 2.5D guide. An example of a 2.5D image artwork model can be seen in [12].

The other type of tactile guides are volumetric guides. These are completely three-dimensional volumetric works that the visually impaired user is able to explore by touching. They are usually the types of guides that gives more information through tactile interaction. However, if the original work is bi-dimensional, it is usually hard or not possible to transform it into a volumetric form. As a result, most of the research related to bi-dimensional art for visually impaired people takes advantage of audio and relief guides, rather than volumetric guides.

In the case of this work, the copy of the artwork is a relief model based on tactile paper embossing technology. The result is a relief guide where only the contours of the features are embossed. This type of thin tactile paper allow us to add thermal interaction so the user can feel the temperature in the fingers while exploring the relief model. This temperature is what communicates depth to the user by way of a novel and intuitive temperature-depth mapping algorithm whose foundation will be explained in the next section.

1.2.5. Temperature-Depth Cross-Modality

Some studies have researched color-concept or color-emotion association. For example, some previous research examined existing beliefs, either subconscious or conscious, about color through color-emotion association by means of an adjective list [13]. Their test consisted of a list of 30 adjectives, randomly ordered, so the users could associate any of the adjectives with any of the colors that were given to them.

While some temperature-concept association research exists, there has not been similar adjective list approaches for inferring the emotional and conceptual responses of the users towards temperature. The existing research focuses on cases such as exploring heat as an expression medium when focusing on interpersonal communication [14] or mapping temperature to the dimensional models of emotion by ratings along valence and arousal dimensions, based in Russell’s circumflex model [15].

However, even with these few pieces of research and approaches about temperature and the conceptual and emotional association that arises from it, there are some results that give a hint about the fact that cold temperatures and warm temperatures are interpreted subjectively as a feeling of remoteness or nearness, respectively. In [16], tests for figuring out subjective interpretations of thermal stimuli in three different scenarios (social media activity, colleague’s presence, and the extent of use of digital content) were performed. The results showed both that there was a strong degree of agreement among participants about what temperature conveyed and that warm temperatures conveyed the presence of a colleague while cool temperatures conveyed absence.

In this work we continue researching in that direction by performing some tests to find out whether users find any correlation between temperature and depth (near–far). These tests, which were performed before the temperature-depth mapping design, are shown in the following section. The results were the basis for the final temperature-depth mapping algorithm.

2. Materials and Methods

2.1. Temperature-Depth Correlation

2.1.1. Temperature Range Used

Before going forward, it is important to define the range of temperatures that will be used from now on, both for the tests and for the design of the algorithms. Our algorithms are designed to help visually impaired people being aware of depth in a more interactive way through thermal interaction. Therefore, the temperature range to use during the tests and for defining the algorithm needs to be fixed from the start. As stated above, the temperature pain threshold is [15, 45 °C] although some research suggest it might be [11, 45 °C] [17], so the temperature range needs to be within those extreme limits. In [5], it was experimentally validated that the temperature range of [14, 38 °C] was convenient and comfortable for the visually impaired users, so it is also the range used in this work.

2.1.2. Temperature-Depth Correlation Test

As has been shown in the background section, temperature-depth correlation has not been extensively researched and only a rough relationship between temperature and the presence or absence of a person has been found. These results give us a glimpse about the possibility of the concepts near/far being somehow correlated to warm/cold temperatures. As a result, our team felt encouraged to figure out whether the hypothesis of depth and distance (near/far) being correlated to temperature might make sense or not.

The test was based, as in [13], on an adjective list for the users to select associations with the warm and cold temperatures. The list had similar adjectives to the ones found in there, with the exception of two added adjectives: the adjectives ‘near’ and ‘far’. The list was made so every adjective had another one with the opposite meaning, creating a list of adjective pairs. The main purpose for having so many adjectives when the research was focused on the depth-temperature relationship was to make the user select adjectives without having any clue that the near/far dichotomy was the one the test was focused on. Two different adjectives lists were prepared: one with all the adjectives ordered randomly (

Table 1. Complete list of adjectives from the test, ordered randomly. Here the user would choose as many adjectives as desired if he/she felt the experienced temperature was somehow correlated to them.

Vivid	Modern
Mysterious	Near
Classical	Sad
Happy	Confidence
Tiring	Dynamic
Depressive	Fearful
Cheerful	Far
Simple	Boring

) and another one with the adjectives ordered in pairs (

Table 2. Complete list of adjectives from the test, ordered in pairs. Also, the “Not Applicable” option was added for each pair. The users were asked to choose one option per row.

Vivid	Boring	Not applicable
Sad	Cheerful	Not applicable
Classical	Modern	Not applicable
Happy	Depressive	Not applicable
Tiring	Dynamic	Not applicable
Confidence	Fearful	Not applicable
Near	Far	Not applicable
Simple	Mysterious	Not applicable

). Each one of the lists was used at a different stage during the test, which will be explain next.

The test was performed twice, first with a total of ten users, five women and five men. The users had an average age of 24 years and were volunteer college students. Each test lasted around 25 min. Then, for verifying whether the findings could be applied to visually impaired people, the same test was carried out again with six visually impaired users. The users were an average age of 17 years and all of them were students from Chungju Sungmo School, a Korean school for visually impaired people. Two of them were nine-year-old kids and all of them where totally blind from birth. Each test lasted around 30 min. The temperature actuators consisted on two petlier devices, a fan, and a heat-sink, which were controlled by an Arduino Mega board. Both temperature actuators can be seen in Figure 1. The test was performed according to the following four steps:

• Step 1

The tester was given an explanation about the procedure of the test and its purpose. However, the testers were told that the main purpose was to find out which concepts were correlated to temperature, without giving any kind of stress or importance to the near/far concepts as to not make the users tempted to select those options on purpose.

• Step 2

After the explanation, the user was asked to touch two different petliers, one which was at a temperature of 15 °C and another one which was at a temperature of 38 °C. After having felt both petliers for as long as desired, they were asked to touch only one of them and to select any number of adjectives that seemed to be related to or conveyed by the temperature they were feeling. Then, the same process was done with the remaining petlier. For each one of the users, the order of the petliers were changed. In other words, if one user started describing the adjectives that suited the warm temperature, then the next user was asked to select the adjectives which were correlated to the cool temperature first.

• Step 3

Finally, the same process was repeated. However, this time the adjectives were purposely ordered in pairs, with the option of ‘Not Applicable’ added to each pair for the cases when the user felt none of both concepts were related to the temperature. This last step was performed for two main reasons: First, the freedom given to the user during the previous step of choosing any adjective from a randomly ordered long list could cause the user to not consider all of them seriously, but rather to just skim through some of them without paying too much attention. On the contrary, the ordered list would force them, at least once more, to go over the near/far adjective dichotomy and consider it in relationship to the felt temperature. Secondly, forcing the user to consider all adjectives and its correlation to the temperatures twice could aid in making the user be more aware of the reason for his/her choices.

• Step 4

Lastly, the testers were asked about the reasons why they chose (or did not choose) the near and far adjectives. Basically, they were asked to justify their answers in order to find out which was the reasoning behind their choice. Also, a brief five-minute conversation about it took place.

2.1.3. Results from Temperature-Depth Correlation Test with the Sighted Users

The results of the tests for the case of the ten sighted users for each one of the two stages can be seen in Figure 2 and Figure 3. The data related only to the near and far adjectives can be seen more clearly in

Table 3. Results during first stage only for the “near” and “far” adjectives.

First Stage	Near	Far
Warm	8 (80%)	2 (20%)
Cold	1 (10%)	3 (30%)

and

Table 4. Results during second stage only for the “near” and “far” adjectives.

Second Stage	Near	Far
Warm	6 (60%)	2 (20%)
Cold	2 (20%)	8 (80%)

, where the number of people that selected each option is indicated next to the percentage in relation to the total number of sighted participants.

2.1.4. Results from Temperature-Depth Correlation Test with the Visually Impaired Users

The results of the tests for the case of the six visually impaired users for each one of the two stages can be seen in Figure 4 and Figure 5. The data related only to the near and far adjectives can be seen more clearly in

Table 5. Results during first stage only for the “near” and “far” adjectives.

First Stage	Near	Far
Warm	5 (83.3%)	0
Cold	0	5 (83.3%)

and

Table 6. Results during second stage only for the “near” and “far” adjectives.

Second Stage	Near	Far
Warm	5 (83.3%)	0
Cold	0	5 (83.3%)

.

2.1.5. Temperature-Depth Correlation Test Results

In general, a correlation between warm and near, and cold and far, can be clearly seen in both stages with both groups. In particular, it is really interesting to see that, in the case of the VIP group, both stages of the test gave quite similar results (totally similar as far as the near/far adjectives are concerned), which indicates that the visually impaired people were probably putting attention and effort during the test since they are more used to, and find more meaning, in trying out different types of interactions. On the contrary, the sighted people seemed to show less interest during the test, which might be the reason why the answers from stage 1 and stage 2 were somehow different. Nevertheless, even in the case of the sighted people, at some point of the test, eight of the ten users agreed that the conceptual dichotomies warm—near and cold—far were correlated. This fact was proven during the test with VIP, where 83% of the users (a total of five out of six) linked in both stages the warm temperature to the concept of being near something, and the cold temperature to the concept of being far. The exception was one of the nine-year-old young kids, who, after hesitating, decided to select the “not applicable” answer, arguing that even though there was some correlation felt, it was not a clear and transparent feeling. This is really interesting since it might not be a matter of chance that the user who had more troubles correlating temperature to the near/far concept was a child. The process which defines temperature-depth association (or temperature and its association to any other concepts whatsoever) is an interesting issue, which should also be taken into account in the future.

2.1.6. Temperature-Depth Correlation Test Interviews

To make sure what their answers and thoughts were, we asked participants the reasons for their choice and what the feeling or thought was that made them choose that particular adjective. The interviews were useful both for finding out the real thoughts of the users and the feeling and thinking behind their answers. In general, every answer considering a correlation between the warm temperature and the concept of “near” were related to the feeling of warmness we feel when we are surrounded by people, when someone is near us. For example, some of the testers said:

“I chose near because I remembered how I feel warm and nice when I am close to people”.

“I chose near because I felt a warm feeling like in a warm atmosphere with people coming towards me”.

However, there were also two users who believed the concept of “far” was better suited to the warm temperature, and they also had their own reasons to feel that. One of the most interesting comments was the following:

“I chose far for the warm temperature because I felt a warm hazy feeling like that of smoke, like distant far away memories”.

Regarding the far–cold correlation, it is interesting that most of the users that chose far for the cold temperature seemed to agree on the reasons. Eight users answered something related to feeling cold because of being lonely and away from people. Some of the exact words were:

“The cold temperature reminded me of winter and of feeling lonely, away from everything”

“The cold temperature made me feel sad and dead, so I felt far away from others and life”

However, there were also answers that justified the opposite: that the cold temperature was a reminder of something being near:

“The cold temperature was felt fast, immediately, like a knife. So it reminded me of something that is really near and true, something that I can easily feel”.

In spite of this differences, it is important to note that most users justified the near–warm and far–cold correlation by sharing similar ideas, feelings and conceptualizations, which in turn, proved that the correlation between farness and cold temperatures, and nearness and warm temperatures can be useful for designing a temperature-depth mapping.

2.2. Temperature-Depth Mapping

The existing correlation between warm and near, and cold and far, was utilized for conveying depth and distance by means of temperature cues. The main idea is simple: the nearer an object is to the user, the warmer the temperature cue. Similarly, the farther an object from the user, the cooler the temperature cue for that object. This idea can be applied in many ways, but we decided to follow a simple mapping method which is explained next, in four steps:

• First step

The temperature range is selected. This can be selected freely (as long as it falls within the comfortable temperature range stated above). In general, the visual perceived distance between the extreme depth levels will be assessed and the extreme temperatures selected accordingly (higher temperature difference for higher distances). However, if the simpler algorithm is to be used, then the extremes depth levels will always be linked to the extreme temperatures of 14 and 38 °C, regardless of their perceived relative distance.

• Second step

The total number of perceived depth levels are counted. For example, in the case of an image with two objects, one in front of the other, there are two depth levels: front and back.

• Third step

The temperature is equally divided in as many temperatures as needed for assigning a temperature to each depth level. The highest temperature and the lowest one are usually assigned to the nearest and the farthest depth level respectively but the use of other initial temperatures might also be possible if is it is considered more appropriate by the designer.

• Fourth step

Optionally, if the difference between the temperatures of two consecutive depth levels is less than 3 °C, some of the consecutive depth levels are clustered together. In other words, some objects from different depth levels are put into a similar intermediate depth level. This helps the user recognizing the different depth levels better by decreasing the total different temperatures to be felt and recognized.

Following these four steps creates a simple mapping that does not consider the absolute distance between depth levels (except for the nearest and farthest depth levels, whose relatively absolute distance is considered for choosing the initial temperature range), but only the number of depth levels and the order in which they approach the user. More complex mappings could be designed, such as a mapping which took into consideration the absolute depth levels of all the features. However, this simple mapping is enough for conveying the different depth levels through temperature cues to give an idea to the VIP of where the different features of the artwork are placed according to depth.

Next, this temperature-depth mapping will be applied and examples of its use will be given in two different types of applications: for representing depth of the different objects of a bi-dimensional artwork and for representing color-based depth of a bi-dimensional image (an effect called chromostereopsis).

2.2.1. Application 1: Artwork Depth

The temperature-depth mapping can be used to convey through temperature the different depth levels of the objects of an artwork. In that way, the visually impaired user can sense more deeply the depth presented in a painting.

In this case, different temperatures for the objects that are at different levels of depth are assigned by following the method presented above.

Before applying the method, first, some techniques used by artists for creating the illusion of depth in visual arts will be contemplated and the method applied to those simple cases. After that, some examples of the temperature-depth mapping method presented above will be applied to two real famous artworks.

Illusion of depth in 2D visual arts

In 2D visual arts, there are many ways of creating the illusion of depth, such as:

-

Overlapping and layering

-

Size and placement and perspective

-

Shading

-

Texture and detail

-

Color, hue, and value

The most relevant ones are layering and overlapping, shading, and size, placement and perspective. However, color, hue, and value can also contribute to create strong feelings of depth, an effect called chromostereopsis [2]. This effect will be explored separately in the following section. First, shading and size, placement and perspective techniques will be explained (since layering and overlapping is a really intuitive and common technique, no explanation will be given).

Shading

Volumetric objects always create shade when being hit by a source of light. As a result, in 2D visual arts, the use of light and shade is one of the methods for creating the illusion of depth. Figure 6 shows an example of an effect called “the crater illusion” [2] in which the central square seems to be in front of the background (right image) or behind the background (left image) depending on the position of highlighted or shadowed edges. The results of applying the depth-temperature algorithm in this case can be seen in

Table 7. Temperature-depth mapping of right side of Figure 6. The central square seems much nearer than the external square, so they are given both extreme temperatures of 38 and 14 °C, respectively.

Color	Layer	Depth Temp. (°C)
Central Square	1	38
Background	2	14

, for the left side of the figure, and in

Table 8. Temperature-depth mapping of left side of Figure 6. The external square seems much nearer than the central square so they are given both extreme temperatures of 38 and 14 °C, respectively.

Color	Layer	Depth Temp. (°C)
Central Square	2	14
Background	1	38

for the right side of the figure.

Size, placement, and perspective

Vertical placement: we perceive objects that are placed lower in the image as closer to us, and objects that are placed higher as being further away. A really clear example of this will be seen later when we apply the temperature-depth mapping algorithm to the artwork “Starry Night” by Vincent Van Gogh.

Diagonal perspective: we perceive diagonal lines as receding into the distance. As shown in Figure 7 and

Table 9. Temperature-depth mapping for Figure 7. The temperature range selected was [26–38 °C], since the external square and the central one seem to be quite far from each other, but not that much far away. The range was then divided by four and each temperature assigned to each one of the depth levels. In this case, basically, the user would feel the temperature decreasing as he/she approached the central square.

Color	Layer	Depth Temp. (°C)
Saturated yellow	1	38
Dull yellow	2	34
Dull red	3	30
Saturated red	4	26

, the red-colored square seems to recede due to the diagonal perspective.

Examples

In this painting, by Matisse (Figure 8), there are seven different elements, which can be seen in Table 8. The dancers have been numbered to aid identification. By looking at the drawing, sighted people can generally agree on five different levels of depth, these levels of depth have been linked to the different depth layers from 1 to 5, with 1 being the nearest layer and 5 being the farthest depth layer from the viewers’ location as a reference. Nearer depth layered elements need to have higher temperatures so temperatures are assigned to the extreme layers (38 °C for the Dancer 5 °C and 14 °C for Sky) and then that temperature range is divided by five (since we have five depth layers). The reason for choosing 38 °C and 14 °C was because Sky and Dancer 5 are visually really far away from each other, so the temperature for those extreme depth layers were chosen in a way that the temperature difference was maximum: the lowest and highest temperatures from the defined [14 °C, 38 °C] temperature range in which we are working. The resulting temperatures are linked to their respective layers and they can all be seen in

Table 10. Temperature-depth mapping of Matisse’s “The Dance”.

Element	Layer	Depth Temp. (°C)
Dancer A	2	32
Dancer B	3	26
Dancer C	3	26
Dancer D	2	33
Dancer E	1	38
Soil	4	20
Sky	5	14

.

In the case of “Starry Night” (Figure 9) by Van Gogh, there are many depth layers, as can be seen in

Table 11. Temperature-depth mapping of Van Gogh’s “Starry Night”.

Element	Layer	Depth Temp. (°C)
Tree	1	38
Village	2	32
Forest	3	26
Mountains	3	26
Stars	4	20
Moon	4	20
Sky	5	14

. As before, these depth layers are selected in a visual way, by contemplating the artwork and choosing the main depth levels defined by the features. In our case, nine depth layers were defined. However, since such a high number of layers would force the user to feel and differentiate many different temperatures, we decided to simplify the number of depth layers. For that, elements were visually and conceptually grouped to check whether some of the elements from different depth levels could be grouped under a common depth level. As a result, the starts and the moon, and the mountain and the forest (both pairs of elements having both conceptual common traits and being visually near to each other) were grouped together in two common depth levels. This can be seen in Table 11 where Forest and Mountains share depth layer number 3, and Stars and Moon share the layer number 4. In this way, the number of temperatures is less and the user can identify the different temperatures and depth layers in an easier way. Therefore, even though technically the forest and the mountains are not in the same depth level, we can simplify it to aid identification. In general, this technique should be performed when the temperature difference between layers becomes less than 3 °C.

2.2.2. Application 2: Chromostereopsis

Another possible application of the temperature-depth algorithm is for conveying the effect of chromostereopsis through temperature. Chromostereopsis [2] is the effect produced by colors on a flat two-dimensional surface by which each color seems to be located in different depth planes, in spite of the two-dimensionality of the image [19]. It is important not to mistake this effect with the association made by artists between red colors and blue colors as advancing and receding colors, since that idea might be based on the brightness produced by atmospheric haze, which is associated with distance, rather than with the effect of chromostereopsis [20]. Chromostereopsis is produced by an effect called chromatic aberration, which is the result from the differential refraction of light depending on its wavelength, causing some light rays to converge before others in the eye and/or to be located on non-corresponding locations of the two eyes during binocular viewing.

Next, an exploratory analysis of the main features that make a color seem farther or nearer will be given, followed by a simple algorithm for conveying chromostereoptic depth by means of temperature. However, first, a brief explanation about colors needs to be given.

Color

The spectrum of color is a continuous one for which there has been several representation models [21]. One of the earliest models is called the Munsell color model, which organized the color perception into a color cylindrical space with three dimensions: hue, chroma (or saturation), and value (or lightness), as can be seen in Figure 10.

Hue refers to the color itself. Luminance means the brightness of a color. The higher the luminance, the closer it is to white, and the lower the luminance, the closer it is to black. Saturation is the vividness (clearness) of a color. As an example, 27 colors of varied hue, luminance, and saturation can be seen in Figure 11.

Chromostereopsis

The chromostereoptic effect is complex and its effects can vary due to many different reasons. Nevertheless, for simple images and in a dark background, red objects tend to appear closer to the observer than blue objects, as can be seen in Figure 12. There, the red and blue stripes will seem to be in separated depth levels for most observers, with the red being apparently nearer to the observer.

This can be extrapolated to warm and cool colors, since, in general (and always when in a black background) warm colors come forward and cool colors retreat [16,19]. However, in [20] researches have also proved that, when the background is white, the effect is reversed, and the warm color seems to be further away than the cool one, as can be seen in Figure 13, which sets the same blue and red colored image to both a black and white background.

An algorithm for conveying the chromostereoptic effect in simple images through temperature cues will be presented. However, first it is necessary to find out why some of these colors seem to recede or advance when in company with other colors. Even though warm colors tend to be felt nearer and cool colors tend to be felt further away, that is not always the effect produced. It seems that the most important features for this chromostereoptic effect in simple images are luminance and saturation.

Luminance

In [20], it was observed that one of the reasons why some colors seemed nearer than others was luminance difference, with bright objects appearing closer than dim ones. This would support the claim that “warm colors tend to advance and cool colors recede” since warm colors tend to be brighter than cool colors. In the following Figure 14, and

Table 12. Temperature-depth mapping of the left side of Figure 14. Even though both colors were visually similar and the depth difference conveyed was not too large, the extreme temperatures of 14ºC and 38ºC were selected but since there are only two depth levels, other temperature difference could have been used, but with the brighter color being warmer.

Color	Layer	Depth Temp. (°C)
Light red	1	38
Dark red	2	14

and

Table 13. Temperature-depth mapping of right side of Figure 14.

Color	Layer	Depth Temp. (°C)
Dark blue	2	14
Light blue	1	38

, the same color can be seen next to each other with different luminance levels. In both cases, the high luminance version of the color seems to be nearer than the dark color.

Saturation

In [24], patches of colored paper against a black background were shown to a total of 17 subjects. In general, they seemed to agree that a desaturation of a color made its depth effect be diminished. This can be seen in Figure 15, where two colors appear at different levels of saturation. In both cases the muted color (the one that is less clear) seems to be farther away. As before, the range of temperature between both colors can be selected freely after assessing the visual depth contrast (similar to applying the first step of the mapping method stated above) as long as the saturated color is the warmest. We chose again the extreme temperatures, as can be seen in

Table 14. Temperature-depth mapping of left side of Figure 15.

Color	Layer	Depth Temp. (°C)
Saturated red	2	38
Muted red	1	14

and

Table 15. Temperature-depth mapping of right side of Figure 15.

Color	Layer	Depth Temp. (°C)
Saturated blue	2	38
Muted blue	1	14

.

Chromostereoptic Temperature-Depth Algorithm

Considering all these features, an algorithm for representing simple chromostereoptic effects with temperature was designed. The algorithm consists of several steps:

• First step

First, the background needs to be chosen (sometimes there is no background or the background color can be simplified and not used, especially when it is a color that is not black nor white). If there is a clear background and it is black or white, it will influence the direction in which the chromostereoptic effect is produced so it is important to take it into consideration.

• Second step

For each color that is not the background, the saturation and luminance level needs to be calculated, summed up, and halved. So, for each color, a value representing its level of saturation and luminance is acquired.

• Third step

The defined temperature range (which is again selected freely through visual relative distance assessment, like was explained in the first version of the temperature-depth mapping method above) is divided by the total number of colors.

• Fourth step

Each one of the temperatures is then assigned to each color in order, according to their luminance-saturation level value. If there is no background or there is a background and it is black, higher luminance-saturation values correspond to higher temperatures; if the background is white, lower luminance-saturation values correspond to lower temperatures.

The algorithm is presented in a more formal and concise way here:

(1)

For each color (except background) => find saturation and luminance level

(2)

For each color (except background) => (saturation + luminance)/2

(3)

Order colors by its luminance-saturation value from highest to lowest into a vector V

(4)

If white background: reverse V.

(5)

Select temperature range and divide it by number of colors.

(6)

For each color in V, assign the temperatures in order from highest to lowest.

As an example, the temperatures of the different colors of two artworks will be shown next.

Examples

In Figure 16 and Figure 17, two artworks of the artist called Mark Rothko can be seen. The temperatures were chosen by following the chromostereopsis-temperature algorithm presented above. For calculating the saturation and luminance level of each color, an app called “Visual Color Picker 2.6”, created by NOVOSIB software co., was used. The saturation (S), lumination (L), and the hexadecimal color code are shown in both

Table 16. Temperature-depth mapping of Figure 14.

Color	Layer	Depth Temp. (°C)
Saturated red (S = 92, L = 92) #EB3812	1	38 °C
Cool red (S = 80, L = 87) #DC2B51	2	26 °C
Saturated blue (S = 82, L = 76) #2373C3	3	14 °C

and

Table 17. Temperature-depth mapping of Figure 15.

Color	Layer	Depth Temp. (°C)
Saturated yellow (S = 95, L = 79) #CA9C09	1	38 °C
Saturated red (S = 73, L = 61) #9E332A	3	14 °C
Saturate orange (S = 81, L = 76) #C47C24	2	26 °C

, where the final depth temperatures of each color is also given. The temperature range selected was the 14 °C and 38 °C. As commented before, this is the temperature range to select when the simplest temperature-depth mapping is desired, one in which the extreme depth levels are always mapped to the extreme temperatures, which users can feel without pain. Similarly, the number of depth layers in each image are three, since those are the number of clearly differentiated colors which contribute to the chromostereoptic effect.

2.3. Thermal Display System Prototype

We designed and developed a thermal display system prototype where the artworks can be installed to feel the different temperatures while exploring them. The system consists of an array of petliers, each one with its own heat-sink and fan, which are driven by a dual H-bridge board controlled through an Arduino Mega microcontroller. The peltier element is a device that releases heat through one side while absorbing heat through the other side when an electric current goes through it. The direction of the current determines which side heats up and which one cools down.

Each petlier from the setup was able to adjust their temperature from as low as 13 °C to as high as 40 °C. The array of petliers can be seen in Figure 18. The artwork is placed on top of the petlier array. The artwork is printed on Thermal Foamed Capsule Paper by means of a braille printer called TactPlus by Kanematsu USA Inc. Tactplus is a printer that uses thermal technology. By heating the paper, braille and graphics can be easily made. As a result, the user can explore the artwork with the hands while feeling the different temperatures. However, for the different temperatures to be felt more clearly, a thin copper layer was placed between the petliers and the artwork. The schematic of this set up can be seen in Figure 19.

Each petlier is set to the desired temperature that the visually impaired user feels with the fingers while exploring that zone of the thermal interactive relief artwork. Mark Rothko’s work was used as the artwork model due to the strong chromostereoptic effect present there and the simplicity of the shapes. However, under the limitation of the array of twelve petliers, any artwork could be installed and the petliers used both for representing either real depth or chromostereoptic depth. An image of the final prototype can be seen in Figure 20.

2.4. Mark Rothko’s Artwork Experiment

To verify the accuracy of the proposed algorithm, a final test was performed. The test was announced through the university bulletin board and, in total, eight college students agreed to form part of the experiment. All of them claimed to be interested in arts and technology. The average age was 26.6 years.

The purpose of this experiment was to test whether visual color depth appearing in Rothko’s work could be transmitted through tactile sense by means of temperature cues, which resulted from the designed temperature-depth mapping algorithm. For this, the relative distance between two objects or colors was defined in a scale from 0 to 5, with 0 meaning no relative distancing at all, and 5 meaning a really strong distance variation between objects. Negative numbers meant the second object was felt farther away than the first. The intention of this test was not notified to the participants until the test was over since the goal was to capture natural feelings through their vision and touch. Therefore, the only information given to the users was the context in which the test was placed (improving art exploration experience for the visually impaired people) and the scale system for defining relative distances that was going to be used during the test.

The test sessions included an explanation of the test and its procedure, visual exploration of the artwork with depth degree scale questionnaire, and tactile exploration of the artwork with temperature conveyed depth degree scale questionnaire. Test duration was about 20 min per person. The testing procedure was the following: (1) Introducing the context of the research, explaining about visually impaired people, art exploration, and thermal cues as a way of presenting different features such as color and depth; (2) Introduction of the petlier display prototype and the artwork installed on it. Also, explanation about depth, color-depth and temperature as a way of conveying nearness and farness; (3) The tester was asked to assess and give a scale degree of the relative distance between color 1 and 2, and color 3 and 2 of the Rothko artwork that can be seen in Figure 21. This assessment was done both visually and though tactile exploration with temperature feedback. The order was reverted for each person, so if one user started exploring the artwork and giving a scale to the relative depth visually, then the next user would start assessing the depth by touching the artwork. While touching the artwork, the petliers under it controlled the temperature of each color.

3. Results

The results of the test can be seen in

Table 18. Results of the final test.

User	V(1 ⇒ 2)	V(3 ⇒ 2)	T(1 ⇒ 2)	T(3 ⇒ 2)	V(1 ⇒2)–T(1 ⇒ 2)	V(3 ⇒ 2)–T(3 ⇒ 2)
P1	−3	+3	−2	+2	−1	+1
P2	+2	+3	−3	+2	+5	+1
P3	−3	+3	−2	+2	−1	+1
P4	−3	+2	−3	+2	0	0
P5	−3	−3	−2	+1	−1	−4
P6	+1	+2	−2	+2	+3	0
P7	+3	+3	−4	+3	+7	0
P8	−2	+2	−3	+2	+1	0
Avg.	−1	1.9	−2.6	2	1.63	−0.13

. V (A ⇒ B) and T (A ⇒ B) indicate the degree of feeling of the difference in depth between colors A and B perceived through visual and tactile sense, respectively. After having all the results, the visual and tactile feeling of the difference in depth between colors were subtracted as a way to measure the difference of depth feeling between the visual and the thermal cues. As an example for understanding the table, the columns related to (3 ⇒ 2) will be explained.

V(3 ⇒ 2) is the visual different of depth created when looking at color 2 after color 3. Because of the chromostereoptic effect, most people find the red to be nearer than the blue and, as a result, the effect is that of looking at a color that is nearer than the first one. This effect is related to the positive scale from [0, +5], with a higher number if the depth difference between both colors is felt higher. Similarly, T(3 ⇒ 2) is the depth sensation created by touching the color that is at 26 °C after the one that is at 14 °C, also graded in the same manner (considering that, as has been proved before, most people feel warmer temperatures as that of something being nearer to us). As a result, we can compare V(3 ⇒ 2) to T(3 ⇒ 2) to assess how similarly or differently the thermal cue allows the user to be aware of the depth difference between two objects or colors compared to the depth difference acquired by the visual sense.

As can be seen in the results, most of the times the difference between the visual and thermal tactile depth feeling was less than one whole scale degree. However, there seem to be some extreme differences in some cases, usually given by the fact that the chromostereoptic effect is not totally universal and a few people seem to perceive the depth levels differently. Such is the case with the participant number seven, who felt that, visually, color 1 was actually farther away than color 2 by a considerable distance.

However, even with this little differences, the average difference between visual and thermal tactile depth cues was less than two whole scale degrees for the comparison between color 1 and color 2, and only −0.1 degree of difference between color 3 and color 2. These are promising results that show that the temperature mapping is a proper translation for depth. Nevertheless, some statistical data analysis can also be performed for narrowing down the confidence interval of the mean difference between visual and temperature cue-based depths. However, since the sample data is not too large, any outlier should be properly spotted and deleted from the sample. For that, first, the median and quartiles of both columns are calculated as in equations 1 and 2. For ease of reading, the column representing V(1 ⇒2)–T(1 ⇒ 2) will be called A, and the column representing the value V(3 ⇒ 2)–T(3 ⇒ 2) will be called B.

$$Q_{A1}=\frac{-1-1}{2}=-0.5\hspace{1em}{Q}_{A2}=\frac{0+1}{2}=0.5\hspace{1em}{Q}_{A3}=\frac{5+3}{2}=4$$

(1)

$$Q_{B1}=\frac{0+0}{2}=0\hspace{1em}{Q}_{B2}=\frac{0+0}{2}=0\hspace{1em}{Q}_{B3}=\frac{1+1}{2}=1$$

(2)

The interquartile range is then calculated:

$$IQ{R}_A=Q_{A3}-Q_{A1}=4.5\hspace{1em}IQ{R}_B=Q_{B3}-Q_{B1}=1$$

(3)

Any value that is below the first quartile or above the third one by an amount of $1.5IQR$ would be considered an outlier. However, in this case there is no outlier so all data needs to be used for the statistical analysis. For calculating the confidence interval of both column A and column B values, the t-distribution is used since not the mean nor the standard deviation of the population are known. Also, other distributions, such as the normal distribution, give better results only when the number of samples exceeds 30.

First, the mean and the standard deviation of columns A and B can be calculated as in (4), where $\mu$ is the mean, $\sigma$ is the standard deviation, $N$ is the number of values, and $x_i$ is each individual value.

$$\mu =\sqrt{\frac{{\displaystyle \sum }{\left(x_i\right)}^{}}{N}}\hspace{1em}\sigma =\sqrt{\frac{{\displaystyle \sum }{\left(x_i-\mu \right)}^2}{N}}$$

(4)

The results after applying those formulas are: ${\mu }_A=1.63$, ${\mu }_B=-0.13$, ${\sigma }_A=3.07$, ${\sigma }_B=1.64$. The confidence interval for the population mean following a t-distribution can be calculated as in (5), where $t_{n-1}$. is the cumulative probability of the t-distribution given a degree of freedom and confidence level, and $n$ is the degrees of freedom calculated as $N-1$.

$$\mu \pm t_{n-1}\frac{\sigma }{\sqrt{n}}$$

(5)

As can be seen in Figure 22, in this case, for a confidence level of 95%, the value of $t_{n-1}$ is 2.365. As a result, the population average of column A and column B falls, with a 95% of confidence, within the range that can be seen in (6). It can be observed that in the case of column B, the difference between the visual and temperature-based depth assessment would not be larger than a scale degree and a half (of the 5-point scale degree that has been defined above). In the case of column A, there is more uncertainty due to the small sample size, but the result is still promising and encouraging for considering temperature-depth mapping and its based temperature interaction for artwork exploration as an interesting option for future assistive devices for the VIP.

$${\mu }_A=\left[-0.9366, 4.1966\right]\hspace{1em}{\mu }_B=\left[-1.5011, 1.2411\right]$$

(6)

In conclusion, the accuracy of the proposed algorithm was verified through the experimental process and the temperature cues were found to be a promising way for conveying the chromostereoptic depth of the artwork. The next step was to install the prototype at an exhibition hall in Chungju Sungmo school for the visually impaired in Korea (Figure 23) to assess the responses of the visually impaired users. The reaction of the users was observed and visitors were briefly interviewed for finding out their impressions about the prototype.

Comments from the visually impaired users at exhibition hall

Visually impaired and sighted visitors were briefly interviewed after using the temperature-depth art prototype exhibited in the Chungju Sungmo School for the blind of Korea. They were asked to comment about their feelings and thoughts regarding the temperature interface and its use for representing depth. In general the responses were positive, with some people stating that it was really interesting to explore the artwork in different ways for which they had not been able to do before. Also, some sighted school teachers pointed about the fact that this kind of new interactions keeps some of the visually impaired people interested in learning since most of the books and tools they use to learn are only braille books or audio recordings. Most of the VIP agreed that temperature was an intuitive way of describing depth because of the correlation between warm and near, and cold and far that we stated above. Also, some of them added, especially the children, that the temperature gave a gamification-like feeling to the prototype, making the artwork exploration more enjoyable and engaging. This statement about the gamification making art exploration engaging and interesting seems to be directly correlated to the teachers’ comments about VIP getting a lot of benefit from new and unusual ways of interaction for keeping up interest by trying out new ways of learning. There were also some complaints, particularly about the fact that in the boundaries of the objects, the temperature from adjacent objects would mix a little bit and the distinction between temperatures was not very clear at those points. Also, some of them suggested that the same system could also be used for adding temperature to some hot or cold objects, such as the sun or water, instead of for representing depth, which could be done instead by adding a third dimension to the tactile model.

4. Conclusions

Throughout this work, two types of tests were performed with sighted and visually impaired users to assess a method to convey depth information by using temperature cues. The first test showed that warm and cold temperatures can be used as cues to communicate to the user how near or far the features of an artwork are. Based on these results, a complete thermal display prototype was designed and developed. Similarly, a relief artwork was designed and installed on top of the prototype, which was used for performing the final test. This final test’s results proved that thermal interaction is a proper way of conveying depth information of the artwork to the VIP. This is an addition to the current technologies which, as was seen above, used to communicate the depth of the features of an artwork either by using audio, or by adding depth into a tactile model by extruding the features. The addition of thermal interaction as a way of communicating depth can open the door to many new ways of experiencing art for VIP. Moreover, the developed thermal display system can be used for adding thermal interaction to any type of paper-based relief artwork, not only by using the thermal cues as a substitute for depth, but also by giving them another role, such as expressing color warmness and color coolness, or for making hot objects (such as the sun) warm and cold objects (such as the water) cold.

Future Work

There are many ways in which this work could be continued or improved on, such as:

• Increasing the number of petliers adding the possibility of creating more complex temperature regions on the artwork;
• Finding a way to make the system smaller and more portable;
• Changing the use of temperature cues from depth representation to an artwork feature semantic mapping, such as making the water feel cold.
• A necessary addition for this semantic mapping would be to be able to make the prototype work with 2.5D relief artworks, which present depth by extruding the features in the z direction, and not only with tactile paper artworks. In that way the visually impaired people could be aware of depth through tactile exploration while also feeling the temperature of the different artwork features while exploring. For that, a method to make the petlier temperature reach all the way to the surface of the 2.5D relief model should be found.

Last Words

In this work, a temperature-depth cross modal mapping for conveying depth in the context of tactile artwork exploration for visually impaired people was designed. The mapping was based in a conceptual and intuitive correlation between temperature and depth. In addition, the developed mapping was applied to two different types of contexts, a complete prototype for adding temperature interaction to paper relief artworks designed and developed, and that same prototype was tested both with sighted and visually impaired users to assess its functionality and the temperature-depth mapping algorithm performance. The results showed that a relationship between depth and temperature exists and that, on the basis of that relationship, depth of artwork features can be transmitted successfully through temperature cues during tactile exploration of an artwork. We hope this work can encourage researchers to consider thermal interaction both as a substitute for depth and as a viable way to improve accessibility for visually impaired people in tactile artwork exploration contexts.