4.3.2. Methodology

The second evaluation was performed in a semi-isolated temporal gallery that we set up within our usability laboratory. The sensory substitution device for visual art color content exploration (Section 3.4) was placed in the gallery. Participants had a ten-minute break between the two evaluations. The test facilitator guided the participants to the gallery. There, following a 2 × 2 Latin Square test design, presented one of the two artwork exploration methods. No color identification tasks were assigned, instead, the participants had ten min to freely explore the visual artwork color contents. Before the exploration task started, the test facilitator provided a short explanation of the exploration method. For the tactile graphic, he also presented a tactile legend with the color-tactile pattern pairs and explained their design. For the sensory substitution prototype, the facilitator explained its function, use, and revised that the participants could trigger the audio and olfactory stimuli using touch gestures. Then, he pointed out the location of the headphones and let the participant explore the artwork independently. When the exploration time was over, the test facilitator conducted a System Usability Survey with the participant. Upon completion, the participant was presented with the other exploration method and performed a similar procedure. Once the participant completed exploration with the two methods, a short unstructured interview was conducted to expand on the information gathered from the survey and obtain feedback for improvements and any personal thoughts or remarks about their exploration experience. The results of this evaluation are described in detail in Section 5.2.

### **5. Results & Discussion**

### *5.1. Multi-Sensory Color Code Effectiveness*

We define the effectiveness of a color code as the color identification success rate achieved by the participants. The color identification success rate is simply the ratio between the number of color identification tasks completed correctly and the total number of identification tasks (eighteen). We computed the rates for both color codes to determine whether the users performed better in color identification tasks using the uni-sensory or multi-sensory approach. We also aggregated the results in different granularities to understand the effect of the several sensory stimuli on the participants' performance. The aggregated results are shown in Figure 3a. Participants were able to identify the color (hue and color dimensions) with a success rate of 69.8% and 71.0% when using the VIVALDI and multi-sensory color codes. In addition, the standard deviation increased from 0.23 to 0.29, which indicates that the multi-sensory color code improved some of the participants' performance, while it hurt others. Thus, from a group perspective, the multi-sensory approach did not perform better in terms of color identification. However, the results for the color identification tasks in Figure 3b show that twelve of the participants (67%) performed equal or better when using the multi-sensory color code. This result is relevant since it suggests that the majority of the participants benefited from the multi-sensory approach. Yet, it also emphasizes the possibility that it might not be an effective solution for everyone. We also calculated the identification success rate for the hue and color dimensions separately. For the hue, the VIVALDI color code had better performance at 82.4% compared to 75.6% of the multi-sensory. Half of the participants experienced a decreased performance when using the multi-sensory approach. Upon further inquiry, we found that some participants reported having experienced cognitive overload during the hue identification

task. For example, one of the participants stated that *"it is important to note that there are some parts where it is difficult to concentrate on music while focusing on scent"*. Another participant also revealed an interesting point that we did not consider during the training session, *"it was difficult to understand the connection between scent and sound"*. Our training session covered the relationship between color-sound and color-scent. However, it did not cover the relationship between sound-scent. After the interview with the participants, we found that they mainly followed two strategies to identify the color during the multi-sensory evaluation. Some of them used a separate approach where they focused on one sensory stimulus at a time and processed the hue and color dimensions identification separately, while others processed the sensory stimuli and identification simultaneously.

For the color dimension identification, the multi-sensory color code performed better with an 89.2% success rate compared to 82.1% achieved with the VIVALDI code. Nine of the participants improved their color dimension performance. Based on the participants' performance, the combined audio and olfactory stimuli of the multi-sensory approach helped some of the participants recognize the color dimensions more easily. One of the participants reported that *"as scent was added to the sound, I (the participant) could feel in more detail and understand more deeply in the process of imagining the color. In addition, it is possible to describe the atmosphere and environment that the music tries to represent in more detail by using the music and scent together"*. Most participants reported that the difference between the color dimensions based on the audio was more subtle than those based on scent. The pronounced difference helped them to identify more easily the color dimension. Regarding the identification success rate for each of the saturated, light, and dark color dimensions, the performance of the multi-sensory approach was higher than that of the VIVALDI code for the first two cases and similar for the last one. For the saturated color dimension, the VIVALDI color code performance was 79.6%, while the multi-sensory achieved 86.1%. In the case of the light dimension, the VIVALDI performance was 74.1%, while the multi-sensory maintained 86.1%. Nevertheless, there was no evidence of statistical difference. However, revising the color identification task performance for the saturated and light color dimensions in Figure 3b reveals that most of the participants improved or maintained their performance using the multi-sensory color code. The participants achieved a 95.4% success rate for the dark dimension for both color codes. Upon hearing the participants' feedback, we found that the high performance achieved when identifying the dark color dimension was because the participants strongly associated the olfactory stimuli of the chocolate scent. While the associations between apple-light and coffee-saturated were not as *"strong"* or *"close"* as reported by the participants. For the VIVALDI color code, the participants stated that both the slow pace of the melody and the low pitch of the notes used also were easy to link to the dark dimensions, while the differences between the pace of the melodies and pitch selected for the saturated and light dimensions where more *"ambiguous"*.

We conducted a short questionnaire to learn how the participants' perceived the color codes' learnability, memorability, and their preference. Ten (55.6%) of the participants expressed that the multi-sensory color code was simpler to learn compared to the VIVALDI color code. Since the multi-sensory color code builds on top of the VIVALDI code, we expected it to be perceived as more complicated to learn. However, the participants expressed that the simple and few associations between the scents and the color dimensions are simple enough. For the memorability of the color codes, eleven participants (61.2%) stated that it was easier to recognize the color from the olfactory stimuli compared to the audio-only stimuli from Vivaldi. One participant considered that *"the relationship (with color) will be remembered longer with the scented one (color code)"* because the scent triggers fond memories. Another participant expressed that it was also the synergy between the audio, smell, and color that allowed her to more easily recognize the color. The last question was about the preference between the two color codes. Thirteen participants (72.2%) preferred the multi-sensory color code over VIVALDI. Among the reasons, one was that they felt the difference between the color dimensions was clearer, and another was that the scents evoked feelings beyond the color space. For example, one of the participants

described feeling *"fresh, springy, and like summer"* during the evaluation. Another participant described the experience as feeling the *"lightness and heaviness"* of the colors. In general, the questionnaire results sugges<sup>t</sup> that the memorability and the experience created by the olfactory stimuli tilted the preference towards the multi-sensory color code.

**Figure 3.** (**a**) Success rate for the different color and color dimensions identification tasks during the VIVALDI and multisensory color codes evaluation. (**b**) Comparison of the participants performance achieved during the evaluation of the VIVALDI and multi-sensory color codes.

#### *5.2. Visual Artwork Color Content Exploration Using a Visual Sensory Substitution Prototype*

The results of the System Usability Survey for the tactile graphic and visual sensory substitution prototype are described in Table 11. In general, most participants (fifteen) gave a better usability score to the multi-sensory-based visual sensory substitution prototype. According to the participants' survey results, the visual sensory substitution prototype scored 78.61, while the tactile graphic method received a 61.53 out of 100. These scores can be interpreted as "acceptable" and "marginal-low" based on the acceptability ranges scores proposed by Bangor et al. [81]. It is important to note that two participants gave a lower overall score to the multi-sensory-based prototype. One of them felt overloaded by the multi-sensory approach. Influenced by this, he obtained the lowest color identification success rate among the group (17% compared to the group average of 71%). Despite

the low performance, the user stated preferring the multi-sensory approach because it considered it more *"aesthetic and supportive to encourage artwork exploration"*. On the other hand, the other participant that gave a lower score had a 100% identification success rate on both exploration methods. This participant's evaluation only focused on the ease of color identification. In her opinion, using the tactile pattern was better *"Since it's more immediate, you don't need to wait for the audio or the smell and as soon as you touch the tactile graphic, you can identify the pattern"*. Thus, there is evidence of the different motivations to explore a visual artwork and how a single approach might be insufficient to cover diverse user needs. Nevertheless, most participants considered that the multi-sensory prototype facilitated the exploration of color contents in visual artworks mainly through its convenient interface, which improved their confidence in identifying the colors and stimulated their imagination experience through its expressive audio and scents.


**Table 11.** Tactile Graphics and Visual Sensory Substitution Prototype System Usability Survey report.

\* SUS score ranges from 1 ("Strongly disagree") to 5 ("Strongly agree").

Participants expressed their inclination to use both methods frequently. Several stated that it would be better if both were available. They could use the visual sensory substitution prototype to explore thoroughly those artworks in which they have an interest and use the tactile graphic to skim through those artworks they do not. Participants did not consider either of the methods to be complex. They said that both were easy to approach, start using, and operate. They deemed the tactile graphics method slightly more complicated because

the pairing between the tactile patterns and color is not intuitive without previous training. The audio and scents were perceived more intuitive, *"(Using the multi-sensory approach) I think even if the color is not evident, I can still feel and experience some sensations from the music like brightness and darkness, or freshness from the smell"*. This characteristic also led the participants to express that they could explore the color contents more independently with the multi-sensory prototype. The opinion was divided in the case of the tactile graphic. Those participants with high tactile proficiency reported a high level of independence were those with low stated the contrary.

Regarding the function integration, the participants gave high scores to both methods. In particular, the multi-sensory prototype was considered well-integrated because of the combined localized audio and olfactory stimuli triggered by touch gestures. Most participants expressed surprise for being able to experience each of the features in the artwork separately through different audio and smell in a similar way as to how visual exploration occurs. However, some touch gesture sensing implementation challenges prevented some of the participants' gestures to trigger the correct stimuli, which lead them to rate the system with some inconsistency. On the other hand, the participants rated the tactile graphic approach as very consistent due to its simplicity. However, we identified that many participants assume responsibility for any problems during the exploration. For example, one of the participants that experienced confusion when touching the patterns said: *"I guess I should practice more (touching) the patterns, I keep forgetting what they mean and the little differences between them"*. The participants described both methods as very easy and straightforward to learn, principally due to the method's simple interface.

The two main advantages of the multi-sensory prototype compared to the tactile graphics concerning usability were its convenience and the confidence felt by the participants. In terms of convenience, the participants' opinion was divided for the tactile graphic approach. Participants that gave a high rating expressed the immediacy of the touch stimuli, the convenience of not having to wear any device. The participants that gave a low rating described the effort to distinguish between the features contour lines from the color patterns and constantly having to review the color pattern legend as very cumbersome. They also pointed out that, while the mental effort to identify the color from the patterns might be the same as for the multi-sensory method, it feels less *"pleasant and artistic and more mechanic"*. The other advantage reported by the participants was the confidence they felt using the multi-sensory prototype in comparison to the tactile graphics. When using the tactile approach, the participants felt difficulty identifying the colors correctly as this approach does not provide other means to confirm the correct identification. The participants felt that the multi-sensory prototype presented them with two alternatives (sound and scent) to perceive the color, making them feel more confident in the identification.

Beyond the identification of color and the usability of the system, the participants communicated that the multi-sensory prototype was better suited to explore and experience visual artworks as the combination of sound and olfactory stimuli encourage exploration and reflection of the artwork. They could not confirm whether their reactions towards the art piece are the same as those intended by the artist, but, in general, they agreed that they had a stronger reaction compared to that experienced when using the tactile graphic.
