2.2.1. Hue

It seems to be a cross-cultural, innate phenomenon that humans do not arbitrarily attach sounds to shapes. The classic example of this is the kiki/bouba effect: when subjects are asked to pick a word that most corresponds to a particular shape, the word kiki is invariably attached to jagged shapes, and bouba to smooth, rounded shapes [30]. One study [31] found that smooth musical timbres were associated with bouba, as were the colors blue, green, and light gray, whereas harsh timbres and the colors red, yellow, and dark gray were associated with kiki. A rounded shape has been associated with sounds from a piano, along with the colors blue and green [31]. In other words, the piano has a rounded tone that corresponds with bouba, so it goes well with green and blue rather than red or yellow. That correspondence is also consistent with Parise's findings that sine waves (soft sounds) are associated with a rounded shape and square waves are associated with sharp angular shapes [32].

Therefore, to represent our six hues, we chose: red, strings (violin + cello); orange, guitar; yellow, brass (trumpet + trombone); green, woodwinds (clarinet + bassoon); blue, piano; and purple, organ.

Red, a warm color that gives the feeling of a hot temperature, is represented by stringed instruments with a passionate tone (violin + cello). Brass instruments convey energy, as if a bright light were expanding, and thus represent yellow (trumpet + trombone). Orange, which is a mixture of yellow and red, is represented by the warm and energetic acoustic guitar. Green, which gives the eyes a comfortable and psychologically stable feeling, is represented by woodwinds with a soft and non-irritating tone (clarinet + bassoon). Blue, a cold color, is represented by the piano, which produces a solid and dense tone with a refreshing feel. Purple, a combination of warm red and cold blue, is represented by the pipe organ, which combines a keyboard with brass tubes.

To express the low, medium, and high values of each color using three pitch levels, a wide range of pitches is required to distinguish between saturated, light, and dark colors. As some instruments such as the violin, trumpet, and clarinet have limited pitch ranges, instruments with similar timbres and different pitch ranges need to be mixed.


**Table 3.** VIVALDI SCC set with wav sound sources.

To listen to a wav file marked with (In Supplementary Materials), left-click the wav audio file and drag it to the computer screen; then right-click to enter the program menu and launch Windows Media Player.

We rearranged the overlapping notes from the original score from Vivaldi's The Four Seasons in Figure 3 with the one shown in Figure 4 such that the harmony can be maintained sufficiently with using the least number of overlapping notes. The reason to use the least number of overlapping notes while maintaining harmony is to highlight the melody. In the two scores shown in Figures 4 and 5, the number of notes differs, and the second score is divided into two parts: high-end and low-end. The high-end instruments, such as the violin, trumpet, and clarinet, use two to three tracks, and the low-end instruments, such as the cello, trombone, and bassoon, use two tracks. The melody is placed on the first track and is played only by the high-end instruments. The low-end instruments play only chords.

In the case of dark colors, playing too many notes at a low volume could make it difficult to hear the notes clearly, so one of the high-frequency chords was excluded for simplicity. Instruments such as the guitar, piano, and organ have wide ranges to express all the required melodies and chords, and thus can represent both high and low values on their own.

Here, the melodies and chords of VIVALDI SCC set (for red, orange, yellow, green) were recorded in a soundproof studio with the collaboration of a sound designer, a composer, and performers. The blue (piano) and purple (organ) sound codes were produced by MIDI.

**Figure 3.** Original track scores for sound coding saturated (left), light (middle), and dark (right) colors from Vivaldi's The Four Seasons.

**Figure 4.** Modified scores for sound coding saturated (left), light (middle), and dark (right) colors played with a violin (same for trumpet and clarinet).

**Figure 5.** Modified scores for sound coding saturated (left), light (middle), and dark (right) colors played with a guitar (same for piano and organ).

### 2.2.2. Value and Chroma

The expressions of value and chroma are based on each season's theme sound. Table 2 shows VIVALDI SCC containing with wav sound source for each color. "Saturated" is from spring, "Light" is from autumn, and "Dark" is from summer. In Vivaldi, each season has entry words. Those for spring are: "Spring has come. Little birds say hello to spring"; those for autumn are: "Villagers rejoice and celebrate the joy of harvesting by dancing and singing"; and those for summer are: "In the season when the sun is strong, men and flocks languish, and the trees and grass are exhausted from the heat." We chose rhythmic melodies to represent saturated, light, and dark while considering those entry words.

High pitched stimuli are generally matched with white and light, highly saturated colors, whereas stimuli from the lower octaves tend to be paired with dark colors [33]. Chroma is expressed through the melody's velocity [16,23–25]; at "saturated," it is expressed as a velocity value of 120 or similar. "Saturated" uses the melody excerpted from spring, and it was judged to be suitable for high chroma by using an A major chord at A4 (middle pitch) to express high chroma. "Light" uses the main theme of autumn, with a high-pitched A5 sound that is played in quick succession and a light atmosphere using F major at A5 (high pitch) to highlight the high-brightness feature. "Dark" uses a slow melody excerpted from summer made dull by progression through E minor at G4 (low pitch) to express a low value. All instrument groups were transposed (compared to the original sound source) to enable actual performance without losing the pitch characteristics of saturated, light, and dark.

### *2.3. CLASSIC SCC V1*

The user feedback on the initial pilot test also motivated us to develop the second melody-based SCC set called "CLASSIC SCC V1" in Table 4 CLASSIC SCC V1 has extracted melodies that match the characteristics of each color from classical music sources which also express the characteristics of the instrument well. The tone, pitch, and intensity of the first note of each melody extracted from a classical music source was chosen to match the characteristics of hue, value, and saturation. The numbers in the table refer to the start time and end time of the extracted sound source. We chose classical music for two reasons. **(1)** In order for the PVI to image the relative position and harmony of the various colors in a piece of artwork, as it will be explained in detail later in Section 2.3, it is necessary to combine the SSCs that express the colors of the picture to produce a well-finished song. Therefore, it is necessary to use sound sources of similar genres as SCCs. **(2)** The word viridi is meaningless to anyone who does not know Latin, but those who have studied Latin immediately think of green. In a similar way, VIVALDI and CLASSIC SCC are color codes expressed by sound, and visually impaired people learn these codes so that they can understand the color composition of the whole picture in a few minutes (e.g., it takes 3 min and 29 s in Table 4). Otherwise, it takes a lot of e ffort and time to read the tactile color pattern, as in [6], even with all 10 fingers, just like reading Braille.

### **Table 4.** CLASSIC SCC V1.

To listen to a wav file marked with (In Supplementary Materials), left-click the wav audio file and drag it to the computer screen; then right-click to enter the program menu and launch Windows Media Player.
