**3. User Tests**

This user test aimed to improve the recognition rate of the proposed SCC by evaluating and analyzing the cognitive accuracy, intuition, and texture of the sound.

### *3.1. Chord Coding Colors (CCC)*

Fifteen college students took part in the exam, which began after participants spent 15 min learning each color corresponding to the sound of an instrument. Two 7" 2-Way Active Studio Monitor speakers were used to play MIDI sound clips by installing RME TotalMix FX v1.50 to control the hardware mixers and effects on an RME audio interface. Each MIDI sound clip was played to the participants, who were asked to identify the color corresponding to each clip. As the participants listened to the sound clips in random order, they were asked to report on the color hue and color attributes (saturated, light, dark) they associated with each sound. The stringed instruments, such as violin, cello, and viola, were difficult to distinguish from one another in the CCC SCC. Likewise, participants found it difficult to distinguish between the horn and oboe, and the oboe and trumpet.

Moreover, low frequency violins and high frequency cellos were difficult to distinguish. On the other hand, the match between yellow and trumpet was easy to remember. The violin (red) was associated with roughness, and the cello (blue) was deemed the softest. The primary color identification rate for each color was as follows: red (violin): 93%; blue (cello): 80%; yellow (trumpet): 93%. However, it was difficult to distinguish the secondary colors: orange (viola) = 53%; purple (horn) = 47%; green (oboe) = 47%.

The subjective feedback from test participants was as follows:

#### *It would be better to express color with a melody.*

*It would be better to use a clearly distinguished instrument rather than a similar instrument such as the oboe and horn.*

### *Mechanically regenerated sound is not natural.*

*If you only hear one sound, it seems di*ffi*cult to distinguish colors. Listening and comparing multiple sounds at the same time could find a connection.*

Thanks to this feedback, we decided that it would be more advantageous to distinguish musical instrument color matching by expressing a melody with a lot of musical personality than to distinguish only by chords.

### *3.2. CLASSIC SCC V1 vs. VIVLADI SCC*

Participants were recruited into two groups: PVI musicians and sighted non-musicians. Eight sighted non-musicians participated in the test. The average age of the sighted participants was 22 years (range 20–25 years). The PVI musicians all had congenital blindness with no color experience. The musicians had all been involved in college-level music studies or had a significant amount of personal education. The non-musicians had not been involved in college-level music research and had no significant amount of personal education.

First, nine congenitally blind musician participants attended the test for CLASSIC SCC V1 and VIVLADI SCC. The average age of the participants was 26 years (range 22–30 years). The participants were given a one-hour tutorial (introduction to color basics and an explanation of how we made the VIVALDI and CLASSIC SCC sets). The sound source and user testimonial questions to be used in the experiment were distributed just before the experiment began. After orientation, participants were presented with sound clips corresponding to 18 colors in each SCC in random order. After listening to each color sound code, the participants evaluated their user experiences. Figure 6 shows the user experience evaluation scores that blind musicians gave the CLASSIC SSC V1 and VIVALDI SCC V1. The overall average scores for the two SCC sets were 3.09 (77%) and 3.47 (87%), respectively. As a result of the paired t-test between CLASSIC SCC v1 and VIVALDI SCC, "ease of use" (t = −4.0, *p* = 0.04) and "texture"(t = −2.63, *p* = 0.03), were statistically significant, but "usefulness" (t = −1.51, *p* = 0.169), "ease of learning" (t = −2.00, *p* = 0.081), and "satisfaction" (t = −1.51, *p* = 0.169) were statistically insignificant. As a result of the analysis, both SCC sets received good scores, but it was found that VIVLADI SCC should improve "sound texture" and CLASSIC SCC should further improve "ease of use." We shall show the improved versions like CLASSIC SCC V2 and VIVLADI SCC V2 in the next sections.

### *3.3. VIVALDI SCC V1 vs. VIVALDI SCC V2*

Table 7 lists the participants' feedback after reviewing the two versions of VIVLADI SCCs. The user experience scores from the eight sighted participants were high (80%) for the VIVLADI SCC V1. As mentioned above, recall that scores from the nine congenitally blind musician participants were even higher (84%) for the VLIVLADI SCC V1. The sound texture of the VIVALDI SCC V1 is annoying because it was created in MIDI. This motivated us to enhance the quality of sound by recording the SCCs in a soundproof studio with the collaboration of a sound designer, composer, and performers. As seen from Table 7, the negative feedback, "It is difficult to distinguish due to differences between instruments," became positive: "It was easy to distinguish and fun because only the instruments were different with the same melody."


**Table 7.** User feedback for VIVALDI SCC V1 and VIVALDI SCC V2.

The PVI with musical experience may be more sensitive to frequency differences and differences in timbre between two signals and may have better short-term auditory memory than blind people with no musical experience [34]. Also, compared with sighted subjects, early blind subjects showed advantages in auditory spectral and temporal resolution, while late blind subjects showed an advantage in temporal resolution [35].

In the negative evaluations other than the positive user feedback in Table 7, non-musicians who evaluated VIVALDI SCC V1 said, "There are many unfamiliar instruments"; "There are instruments that are difficult to discern"; and "Color transmission is difficult." Other said, "It's difficult without appreciating art at the same time"; "In most cases, the sound mood doesn't match the chosen color." However, congenitally blind musician participants who participated in the VIVALDI SCC V2 evaluation provided all positive feedback.

The participant who provided negative feedback thought that in the case of a congenitally blind person without color experience, the color name would have a completely different meaning from what is visible, so it is sufficient to convert the image itself into a melody directly without converting it to a melody corresponding to the color of the image. This method has the advantage of enhancing the expressive power of one work, but also has the disadvantage that it cannot be applied consistently to other works.

Moreover, representing the image itself in sound is a much more difficult and complex process than using color names. The feeling that comes from an image or color is very subjective, so deciding which image to express with some sound is very difficult. Therefore, the current method of linking the characteristics of an instrument to the characteristics of color is objective and easy to access. Just as sighted people learn color names, blind people need to learn not only color names, but also musical codes that correspond to colors.

### *3.4. CLASSIC SCC V1 vs. CLASSIC SCC V2*

Table 8 lists the user feedback for the CLASSIC SCC V2 after it had been updated to reflect the user feedback from the test results of CLASSIC SCC V1. User experience scores from the eight sighted participants averaged 79% for the CLASSIC SCC V2. As mentioned above, recall that scores from the nine congenitally blind musician participants were even higher (84%) for the CLASSIC SCC V1.


**Table 8.** User feedback for CLASSIC SCC V1 and CLASSIC SCC V2.

In addition to the feedback on CLASSIC SCC V1 in Table 8, we go<sup>t</sup> the following: "It is difficult to change the pitch range for each instrument because a specific pitch range is not established"; "Due to the variety of instrumental accompaniment, the color of the instrument is not clear." To solve this problem as a whole, we replaced the red, orange, and blue sound sources of CLASSIC SCC V1, Table 4, with the sound sources of CLASSIC SCC V2, Table 5.

There was negative feedback about the CLASSIC SCC V2: "The shorter the length of the sound segmen<sup>t</sup> representing each color, the better you can remember and distinguish S, L and D." That is true, but in this case, when creating music that represents the color composition of the entire artwork, we should make sure that each segmen<sup>t</sup> is not too short in length to seamlessly connect the sound segments corresponding to each color.

### *3.5. VIVLADI SCC V2 vs. CLASSIC SCC V2*

Three male participants (26 years old on average) with congenital blindness and no experience of color attended the test, in which we wanted to learn how easily the sounds of the VIVALDI SCC V2 and CLASSIC SCC V2 could be identified. Among them, two participants were musicians who played the trombone and guitar, respectively, in a chamber orchestra. After a 30 min tutorial orientation for both SCCs, the subjects were asked to listen to the sound sources threre times to familiarize themselves with the VIVALDI SCC V2. They were then asked to listen to the VIVALDI SCC V2's 18 color sound codes in a random order, and they were asked to choose one of three color attributes (hue, saturation, or light and dark) that corresponded to each code. Next, the same procedure was used to test the CLASSIC SCC V2. Both SCCs scored 100% on the identification test.

#### *3.6. Vincent van Gogh's The Starry Night with CLASSIC SCC V2*

The user experience score evaluated from eight sighted students for artwork music composition of "Vincent van Gogh's The Starry Night" applied with CLASSIC SCC V2 was 83.5%.

The feedback on the composition of Vincent van Gogh's The Starry Night has all been very positive as follows:

*The purpose is clearly necessary and useful in expressing color vision through hearing. I really want to recommend it.*

*It takes a lot of imagination.*

*It is easy to remember because it can remind us of a rough expression of color in your head. If the rules of the code are maintained but composed of various melodies, the beauty of the work will be conveyed well. It also shows variety.*

*It is clear and simple, but even looking at one piece of artwork, there seem to be many people making di*ff*erent music with similar pitches.*

*The division of each part of the artwork was expressed with cymbals, and it was good to appreciate. When it was arranged with the image of Van Gogh, it was clearer because it was suitable music. I am satisfied because it goes well with art, and I like music*.
