**4. Experimentation**

Inspired by the research related to music and color association [21], we chose to use a common semantic and emotive link to compare the different modalities. We used as a link a list of adjectives and the emotional-semantic space they created. Osgood [22] simplified the semantic space of adjectives into three aspects, which are (1) evaluation (like–dislike), (2) potency (strong–weak), and (3) activity (fast–slow). The adjectives we decided to adopt in this research are the pairs of adjectives that people are familiar with; emotion, shape, location, activity, texture, contrast, temperature, sound characteristics, and so on.

As a result, 18 pairs of adjectives, each set of six representing one of the three different Osgood semantic spaces, were chosen. In other words, there was a set of adjectives representative of the whole semantic space with which to create a comparison between modalities. That modified list of adjectives from Osgood [22] can be seen in Table 6, where the column indicates the Osgood adjective list.

The main idea for the test was the following: classifying the adjectives into warm/cold and dark/bright dimensions, correlating the different adjectives to the different sound and temperature cues, and, as a result, giving to each sound and temperature a warm/cold and dark/bright score to find out which cues expressed the warm, cold, dark or bright dimensions the best. Additionally, those scores would help us find out whether VIVALDI or CLASSIC was a better option for representing the color dimensions and whether it was better to express color hue with temperature and color dimension through sounds or vice versa.


**Table 6.** Selected semantic adjectives (modified from Osgood [22]) to be used as a common link between sound coding colors and color dimensions like warm/cold and bright/dark.

> In total, three different types of tests were performed: one for classifying the different adjectives into the warm/cold and dark/bright main dimensions, a second one for finding out the weights of each particular pair of adjectives with the dimensions they were classified into, and one last test which had the users selecting adjectives for each one of the temperatures and sound cues. The three test were analyzed and the results presented in the analysis section.

> The number of participants was 18. They were college students who had normal eyesight and an average age of 21.5 years old. Since tests were performed on different days, not all of them were able to participate in all tests. 15 users participated in the first two tests and a total of 12 users in the third one. All the test sessions included an explanation of the test and its procedure before starting the surveys. Test duration varied, with the first and second one lasting together for about 25 min per person and the last test lasting for around 45 min per person. The testing procedure was the following:


### *4.1. First Test: Classifying Pair of Adjectives*

Users were asked to select which color dimension was related to each pair of adjectives the most: the bright/dark dimensions, the warm/cold dimensions, both of them or none. As an example, the answer of one of the users for the adjective pair "noisy–quiet" can be seen in Table 7. There, the user stated that the dimensions that is more correlated to the "noisy–quiet" adjective pair was the bright/dark dimension of the colors. Table 8 presents the results after summing up all the answers. The number indicates the total of users that ticked an option. Since some users selected the option "both", it is possible that the sum of the bright/dark and warm/cold counts results in a number higher than that of the total number of users.


**Table 7.** Example of the dimension survey for the adjective pair "noisy-quiet".

**Table 8.** Total count of answers for all adjectives after surveying 15 participants. L/D is the bright/dark dimension and W/C is the warm/cold dimension.


#### *4.2. Second Test: Calculating Weights of Each Pair of Adjectives for Each Dimension*

In this test, the users were asked to rate the correlation of each pair of adjectives with each individual dimension: warm, cold, bright, and dark. The scale used was from −2 to 2. Table 9 shows the answer of one of the users for the adjective pair "noisy-quiet" in relation to the color dimension "bright".

**Table 9.** Example of the weight score survey answered by one participant for the dimension bright and the adjective pair "noisy–quiet". Participants were asked to give a weight score of each pair of adjectives for each one of each color dimension.


Once all the users rated all the adjective pairs in all the dimensions, four tables with the weights for each dimension and pair of adjectives, graded from −2 to +2, were made. Negative numbers indicate that the dimension is directly correlated with the adjective from the left column, while positive numbers indicate that it is directly correlated with the adjective from the right column. The number indicates how strongly the correlation is, with −2 and +2 being the strongest correlation and a zero meaning there is no correlation at all. As an example, in Table 10, the weights for the dimension of bright can be seen. Similar tables for warm, cold, and dark dimensions were also made but omitted here for brevity.


**Table 10.** All weight scores for each adjective pair or the "bright" dimension. Similar weight tables were acquired for the warm, cold and dark dimensions. The standard deviation is indicated next to the value in parenthesis.

#### *4.3. Final Test: Linking Adjectives to Each Modality Cue*

During the final test, all modalities and all cues (VIVALDI sound, CLASSIC sound, and temperatures) were given a score for each one of the pairs of adjectives stated above. For example, in the case of temperature modality, the process was the following.

First, the user would feel one of the temperatures seen in Table 2, and, for each one of those temperatures, a form sheet like the one shown in Table 11 would be filled up. As an example, Table 11 has been filled up with all the answers of one of the users after having felt the 38 ◦C temperature Peltier. This same process was performed not only with the rest of the temperatures but also after listening to each one of the sounds from the VIVALDI and CLASSIC sounds. As a result, there was an adjective-graded sheet for each one of the cues (each one of the temperatures and sounds) of the three different modalities contemplated.

**Table 11.** Adjective score table of the 38 ◦C temperature cue filled up with the answers of one of the participants. The users answered similar score tables after hearing to each musical sound and feeling each temperature.


From all the testers' answers, an average score on the scale of [ −2, 2] was calculated. The value of −2 would be the equivalent to all testers giving a score of 1 to the adjective during the survey. A score of +2 would be the result of all participants giving a score of 5. As an example, the results for the saturated red of the VIVALDI set of sounds can be seen in Table 12.

**Table 12.** Adjective score results for VIVALDI red saturated color. Similar adjective scores were acquired for all colors and temperatures from the third test.


### **5. Analysis and Results**

*5.1. Analysis*

Analysis of the results was performed for finding out the best design of the multisensory color-mapping system. The analysis process was the following.

First, by means of the results presented during the second test, we created a weight table indicating all the pairs of adjectives and their relative weights to each one of the four dimensions: bright, warm, cold, and dark (Table 13). However, only the adjectives that were selected during the first test as related to that particular dimension were taken into account when filling up the table. Therefore, empty weights are the set of adjectives that were uncorrelated to that particular dimension on the first test's results. Both the weight table and the adjective score (like the one shown in Table 12) for each one of the sounds and temperature cues were used to calculate a bright score, dark score, warm score, and cold score for each cue. The basic formula for each of those four scores was the following.

$$S\_{cd} = \sum\_{i} (\mathcal{W}\_{di} + \mathcal{S}\_{ci}) \tag{1}$$

where *Scd* is the total score of the dimension "*d*" for the cue "*c*", *Wdi* is the weight of the dimension "*d*" for the adjective pair "*i*", and *Sci* is the score of the adjective pair "*i*" for the cue "*c*".


**Table 13.** Weight score summary of each adjective pair for each one of the four dimensions.

After applying the formula for all dimensions and each sound and temperature cue, three color dimension score tables were created, one for VIVALDI and all its sounds, other one for CLASSIC and all its sounds, and the last one for all the temperatures. An example of all the scores for the VIVALDI set can be seen in Table 14. Only the positive results are relevant, since it indicates cues that are directly correlated to the different dimensions. In Figures 2–5, bar plots graph indicating all the positive scores of all sets for each dimension can be seen. In Figure 6, the highest store per dimension for each set is presented. Lastly, in Figures 7–9, all the scores of all the cues (negative values included) are presented in a boxplot graph for each one of the methods and dimensions.



**Figure 2.** Positive scored cues in all sets for the bright dimension.

**Figure 3.** Positive scored cues in all sets for the dark dimension.

**Figure 4.** Positive scored cues in all sets for the warm dimension.

**Figure 5.** Positive scored cues in all sets for the cold dimension.

**Figure 6.** Highest score graph of the three methods for each of the four dimensions.

**Figure 7.** Boxplot of the four dimensions for all points for VIVALDI. The average, max., and min. values, together with the different percentiles, can be seen.

**Figure 8.** Boxplot of the four dimensions for all points for CLASSIC. The average, max., and min. values, together with the different percentiles, can be seen.

**Figure 9.** Boxplot of the four dimensions for all points for TEMPERATURE. The average, max., and min. value, together with the different percentiles, can be seen.
