**1. Introduction**

Artists believe that art is central to human life. Unfortunately, many people with visual impairments (PVI) do not have access to the world's visual culture or the opportunity to experience the life-enhancing power of visual art. PVI must have access to the world's visual culture if they are to participate fully in their communities and the world at large. Such access will improve the quality of their lives and help them gain skills crucial to their education and employment opportunities [1]. Findings from participant observations in touch tours for blind and visually impaired people at the Metropolitan Museum of Art were discussed by Hayhoe [2]. Touch Graphics Inc. exhibits a reproduced painting called "Talking Tactile" at the San Diego Museum of Art [3]. Recently, "BlindTouch" [4,5] has been introduced to enhance PVI's artwork experience by reproducing painting masterpieces as 3D-printed models that contain touch recognition sensors (based on conductive painting) and provide relevant audio descriptions and sound e ffects (such as the sound of the wind or a flying butterfly) while the user explores the di fferent features of the artwork with their fingers. The touch interface was evaluated by PVI and found to be e ffective. The audio feedback can be activated by tapping the fingers. However, PVI still find it di fficult to e ffectively experience color when appreciating a museum's artwork. Tactile color pictograms (TCPs), which are embossed on a surface that PVI can touch to perceive color information, are used to allow easy accessibility to color information. TCPs have two advantages as an assistive tool when used in conjunction with audio descriptions. First, they allow immediate access to color information through color patterns, just as a sighted person sees colors

immediately. Second, embossing a TCP directly on a piece of artwork reproduction surface allows PVI to grasp color and color-related information (e.g., shape, size, brightness, and position) through tactile interaction [6].

However, for color recognition, tactile patterns coding colors (TPCs) alone might not provide a good user experience for PVI (especially those with congenital blindness or complex disabilities) because tactile interactions tend to be slow. Congenitally blind people understand colors through physical and abstract associations. Color audition means the reaction of feeling color in one sound. Gauguin tried to pursue a symbol of innerness as a color. He said, "Color which, like music, is a matter of vibrations, reaches what is most general and therefore most indefinable in nature: its inner power" [7].

Therefore, it might be desirable to provide color images for PVI that convert the color being touched into a corresponding musical sound, such as orchestral or classical music, that codes the color. In other words, to provide better color perception for PVI, a multisensory user experience that combines tactile and musical stimuli might be e ffective. A combination of TPCs and sound coding colors (SCCs) could also enable PVI to interpret the overall color composition of a piece of artwork. Therefore, we here explore the association of sound tones (timbre, pitch, and intensity) with the color properties (hue, value, and chroma) of the Munsell color system. Based on the observations and series of user tests performed in this research, we here introduce two SCC sets that represent 18 colors in 6 hues.

### *1.1. Review of the Color System*

The Munsell color system is a color space that specifies colors based on the three properties of hue (basic color), value (lightness), and chroma (color intensity) [8]. In this system, the higher the lightness value, the closer the color is to white, and the lower the value, the closer it is to black. Chroma is the vividness (clearness) of a color. Colors with the highest chroma are pure colors (saturated colors), meaning that they are not mixed with other colors, and colors with the lowest chroma are achromatic (white, gray, black). Palmer et al. described 37 colors that reflect the color preferences of US college students [9]. Munsell's hue/value/chroma data from those 37 colors can be found in [9]. The SCC sets proposed in this paper can express six unique hues (red (R), orange (O), yellow (Y), green (G), blue (B), and purple (P)) using three color dimensions—saturated (S), light (L), and dark (D)—for each hue, as shown in Figure 1.

**Figure 1.** Red (R), orange (O), yellow (Y), green (G), blue (B), and purple (P) using three color dimensions—saturated (S), light (L), and dark (D).

For example, in Figure 2, the color marked "Light" has a value of seven and chroma of eight. The color marked "Dark" has a value of three and a chroma of eight. The color marked "Saturated" has a value of five and a chroma of 15. The colors with the lowest chroma (0) are achromatic.

**Figure 2.** Saturated (S), light (L), and dark (D) for red in [9].

### *1.2. Review of the Sound Representations of Colors*

The perception of a sound has several aspects:

Pitch: the frequency of a sound (high or low).

Tone: the timbre of a pitch.

Key: is it a major or minor scale?

Timbre: the characteristic sound of an instrument (e.g., a flute and an oboe playing the same note sound differently).

Chords: two or more notes played at the same time.

Melody: sequentially arranged notes.

Volume: loudness.

#### Velocity (the musical term): **the force with which a note is played, and it is vitally important in making MIDI performances sound human.**

Synesthesia, a mixing of the senses, occurs in some individuals. People with strong synesthesia, synesthetes, experience a perception in one sense when a stimulus for another sense is presented (e.g., seeing a color when hearing a sound). This is what Martino and Marks [10] call "strong synesthesia" as opposed to "weak synesthesia," which describes the simple cross-sensory correspondences that most people experience, e.g., being able to match a tone of a certain pitch to a light of a certain brightness [11]. One of the most common types of synesthesia is seeing color while listening to music or musical notes (e.g., [12–15]). Gaboski and Odebert [12] showed some tendency to associate color with short musical selections. Slow music is associated with blue, high tones are light, and low tones are dark. Fast and cheerful notes are reminiscent of bright and intense warm colors.

As a synesthete, Kandinsky saw colors corresponding to the sounds of different instruments and used those correspondences in his paintings [13]. He attributed musicality to his paintings and said that painting could generate energy like music, "drawing is rhyming the form with color and showing the moving power through color." He believed that just as musicians express their feelings in musical forms such as rhythm, timbre, and melody, artists can express their inner experiences of fear, sadness, and joy using various arrangements of color and form. He associated colors and forms with specific emotions, connected them with elements of music, and then sought to write those subjective and emotional experiences into logical and universal laws. "Yellow is stimulus, red is energy, blue is infinite sensibility, and green is calm," said Kandinsky.

People without synesthesia also report correspondences between music and color. For example, [14,16] found some evidence that music–color correspondences result from an emotional

link. Brighter colors such as yellow, red, green, and blue were usually assigned to happy songs, and gray was usually assigned to sad songs [17]. A study by Palmer [9] also suggested that fast notes in a major key are yellow or orange and slow notes in a major key are blue and gray. He found cross-modal correlations based on the associative sensibility between color and music with the help of music that involves emotion.

Newton's *Opticks* [18] showed that the colors of the spectrum and the pitches of musical scales are similar (for example, "red" and "C"; "green" and "A*b*"). Maryon [19] also explored the similarity between the ratio of each tone to the wavelength of each color to connect them. This method of associating the pitch frequency of the scale with color can be a way of substituting colors and notes for one another [20]. However, the various sensibilities that can be obtained through color are limited by simply substituting colors into the musical scale. Alber Lavigna [21] found that the technique of a composer in organizing an orchestra seems very similar to the technique of a painter applying colors. In other words, a musician's palette is a list of orchestral instruments.

A comprehensive survey of associations between color and sound can be found in [22], including how di fferent color properties such as value and hue are mapped onto acoustic properties such as pitch and loudness. Using an implicit associations test, those researchers [22] confirmed the following cross-modal correspondences between visual and acoustic features. Pitch was associated with color lightness, whereas loudness mapped onto greater visual saliency. The associations between vowels and colors are mediated by di fferences in the overall balance of low- and high-frequency energy in the spectrum rather than by vowel identity as such. The hue of colors with the same luminance and saturation was not associated with any of the tested acoustic features, except for a weak preference to match higher pitch with blue (vs. yellow). In other research, high loudness was associated with orange/yellow rather than blue, and high pitch was associated with yellow rather than blue [23].

Chroma has a relationship with sound intensity [23,24]. When the intensity of a sound is strong and loud, its color is close, intense, and deep. However, when the sound intensity is weak, the color feels pale, faint, and far away. Higher value is associated with higher pitch [17,25]. Children of all ages and adults matched pitch to value and loudness to chroma. The value (i.e., lightness) is high and heavy dependent on the light and dark levels of the color. Using the same concept in music, sound is divided into light and heavy feelings according to the high and low octaves of a scale. When the intensity of the sound is strong, the color sensed is close and sharp, whereas when the intensity of the sound is weak, the color becomes distant and muted [26].

Another way to match color and sound is to associate an instrument's tone with color, as in Kandinsky [14]. A low-pitched cello has a low-brightness dark blue color; a violin or trumpet-like instrument with a sharp tone feels red or yellow; and a high-pitched flute feels like a bright and saturated sky blue. As shown in Table 1, it is possible to compare the sensibility felt in each instrument tone with the sensibility felt in color.

In "SeeColor" [27], when you touch a relief-shaped, embossed outline area, the color associated with that area is transmitted to an instrument's sound.

In color sonification [28], the color hue, chroma, and value attributes of pixels are mapped onto sound parameters as the f0 (which is related to a sound's perceived pitch), the spectral envelope of the sound (which influences the perception of timbre), and the intensity (which is related to the sound's perceived loudness), respectively. The sound output of a hue value is thus a sinusoid whose frequency depends on that value. The lower frequency colors, such as red and orange, give sensations of strength and power, so they relate well with higher pitched sounds.

The future work from [28] is to deal with more complex sounds and rhythmic patterns. However, there was no assessment provided from these methods to confirm the sound-code literacy of PVI.

In this paper, based on the aforementioned observations, we introduce two SCC sets, VIVALDI and CLASSIC, produced with rhythmic instrumental sounds of classical melody. Table 1 shows the previously proposed color–instrument matchings and the two SCC sets proposed by us.


### **Table 1.** Existing SCCs with instruments and ours.

### **2. Materials, Methods**

### *2.1. Chord Coding Colors (CCC)*

The purpose of this study was to create sounds by which the colors in an image are expressed. Our first SCC set is a so-called chord coding colors (CCC) set as shown in Table 2. Each color's chroma and value are represented by a unique musical chord (e.g., the G chord is made up of the notes G, B, and D. The Gm chord is comprised of G, Bb, and D) that sounds as a single note. In the CCC set, the colors red, orange, yellow, blue, green, indigo, and purple are represented by the instruments violin, viola, trumpet, oboe, cello, horn, and saxophone, respectively. The color–instrument assignment mostly stems from Kandinsky in Table 1. High chroma (saturated) maps to a sound intensity denoted by "ˆ" (velocity = 127), and the medium chroma light and dark map to sound intensity (velocity = 60). A high value (light) maps to a high pitch, denoted by "H" (C4 to C6), and a low value (dark) maps to a low pitch, denoted by "L" (C2 to C4). The codes between colors are all the same, so there is no difficulty to understanding saturated, light, and dark.




**Table 2.** *Cont.*

To listen to a wav file marked with (In Supplementary Materials), left-click the wav audio file and drag it to the computer screen; then right-click to enter the program menu and launch Windows Media Player.

### *2.2. VIVALDI SCC*

The user feedback on the initial pilot test motivated us to develop our two melody-based SCC sets (Tables 3–5). Our VIVALDI SCC was inspired by Vivaldi's The Four Seasons, which musically represents elements of the four seasons of the year. We extracted some of Vivaldi's melodies from spring, autumn, and summer that matched the characteristics of saturated, light, and dark colors, respectively. The most important part in matching instruments and colors is avoiding similar tones so that the sounds (colors) are clearly distinguishable. Therefore, to express each color, we used two strings, two wind instruments, and two percussion instruments for easy identification. In addition, the composition of the excerpted part was changed from original sound, and the velocity and pitch were adjusted to clarify the distinction between value and chroma.
