*2.7. Hearing (Sound)*

Hopkin [66] confirmed that congenital blind or who lost their sight during the first two years of life do indeed recognize changes in pitch more precisely than sighted people. However, there were no significant differences in performance between sighted people and people who had lost their sight after their first two years of life. These findings reveal the brain's capacity to reorganize itself early in life. At birth, the brain's centers for vision, hearing, and other senses are all connected. Those connections are gradually eliminated during normal development, but they might be preserved and used in the early blind to process sounds.

The Metropolitan Museum of Art in New York introduced the reproductions of soundsensitive art objects by attaching sound switches [67]. The switch plates were cut into shapes based on the form of the major elements of the painting. When someone touches a particular element, an ambient sound related to that element of the painting is produced.

The sense of immersion is improved for the viewer when an artwork is experienced using more than one sense [68,69]. Visual images affect the sensibility of the viewer, conveying meaning, and sound affects the sensibility of the listener. Thus, the effect of a visual image can be maximized by harmonizing the sensibility of the visual image with the ambient sounds. Research has shown that pairing music and visual art enhances the emotional experience of the participant [70]. In the "Feeling Vincent Van Gogh" exhibition [71], a variety of interactive elements were used to communicate artworks to viewers, who could see, hear, and touch Van Gogh's works and thus appreciate them through multiple senses. Visitors could feel Van Gogh's brush strokes on 3D reproductions of Sunflowers and listen to a fragment of background sound through an audio guide [71]. The experience was intended to stimulate a deep understanding of the work and provide a rich imaginative experience. "Carrières de Lumières" in Levod Provence, France, and "Bunker de Lumières" in Jeju, Korea [72], are immersive media art exhibitions that allow visitors to appreciate works through light and music, providing an experience of immersing in art beyond sight.

Every moment of seeing, hearing, and feeling an object or environment generates emotions, which appear intuitively and immediately upon receiving sensory stimulation. Sensibility is thus closely related to the five senses, of which the visual and auditory are most important. Among sighted, hearing people, information from the outside is accepted in the proportions of 60% visual; 20% auditory; and 20% touch, taste, and smell together [73].

Sound can work together with sight to create emotion, allowing viewers to immerse themselves in a space. Therefore, Jeong et al. [74] designed a soundscape of visual and auditory interactions using music that matches paintings to induce interest and imagination in visitors who are appreciating the artwork. That study connected painting and music using a deep-learning matching solution to improve the accessibility of art appreciation and construct a soundscape of auditory interactions that promote appreciation of a painting. The multimodal evaluation provided an evaluation index to measure new user experiences when designing other multisensory artworks. The evaluation results showed that the background music previously used in the exhibit and the music selected by the deeplearning algorithm were somewhat equal. Using deep-learning technology to match paintings and music offers direction and guidelines for soundscape design, and it provides a new, rich aesthetic experience beyond vision. In addition, the technical results of that study were applied to a 3D-printed tactile picture, and then 10 visually impaired test participants were evaluated in their appreciation of the artworks [74].

#### **3. Coding Colors through Sound, Pictograms, Temperature, and Vibration**

According to Merleau-Ponty (1945/2002), color was not originally used to show the properties of known objects, but to express different feelings suddenly emerging from objects. According to Jean-Paul Sartre's aesthetics of absence, art is to lead to the world of imagination through self-realization and de-realization of the world. Aesthetic pleasure is caused by hidden impractical objects. What is real is the result of brushing, the thick layer of paint on the canvas, the roughness of the surface, and the varnish rubbed over the paint, which is not subject to aesthetic evaluation. The reason for feeling beauty is not mimesis, color, or form. What is real is never beautiful, and beauty is a value that can only be applied to the imaginary. Absence is a subject that transcends the world toward the imaginary. When reading the artist's work, the viewer feels superior freedom and subjectivity. According to the theory of perception, viewers give meaning to the work according to their experiences. Color is not an objective attribute, but a matter of perception that exists in the mind of the perceiver. It is also known that emotions related to color are highly dependent on individual preferences for the color and past experiences. Therefore, color has historically and socially formed images, and these symbols are imprinted in our minds, and when we see a color, we naturally associate the image and symbol of that color. For example, we can look at such embodied images [75] of color in Vincent van Gogh's work. Vincent Van Gogh went to Arles in February 1888 in search of sunlight. There he gradually fell in love with the yellow color. His signature yellow color is evident in his vase with fourteen sunflowers. Gogh was drawn to the yellow color of the sunflower, which represents warmth, friendship, and sunlight. He said to himself, "I try to draw myself by using various colors at will, rather than trying to draw exactly what I see with my eyes." On the other hand, according to Goethe's Color theory, yellow has a bright nature from purity, giving a pleasant, cheerful, colorful, and soft feeling.

Synesthesia is a transition between senses in which one sense triggers another. When one sensation is lost, the other sensations not only compensate for the loss, but the two sensations are synergistic by adding another sensation to one [76]. For example, sight and sound intermingle. Music causes a brilliant vision of shapes, numbers and letters appear as colors. Weak synesthesia refers to the recognition of similarities or correspondences across different domains of sensory, affective, or cognitive experience–for example, the similarity between increasingly high-pitched sounds and increasingly bright lights (auditory pitchvisual color lightness). Strong synesthesia, in contrast, refers to the actual arousal of

experiences in another domain, as when musical notes evoke colors [76]. Synesthesia artists paint their multi-sensory experiences. Vincent van Gogh's work is known for being full of lively and expressive movements, but his unique style must have a reason. Many art historians believe that Vincent van Gogh has a form of synesthesia, the sense of color. This is a sensational experience in which a person associates sound with color. This is evident in the various letters Van Gogh wrote to his brother. He said, "Some artists have tense hands in their paintings, which makes them sound peculiar to violins", he said. Van Gogh also started playing the piano in 1885, but he had a hard time holding the instrument. He declared that the playing experience was overwhelming, as each note evokes a different color.

The core of an artwork is its spirit, but grasping that spirit requires a medium that can be perceived not only by the one sense intended, but also through various senses. In other words, the human brain creates an image by integrating multiple nonvisual senses and using a matching process with previously stored images to find and store new things through association. So-called intuition thus appears mostly in synesthesia. To understand as much reality as possible, it is necessary to experience reality in as many forms as possible, so synesthesia offers a richer reality experience than the separate senses, and that can generate unusually strong memories. For example, a method for expressing colors through multiple senses could be developed.

The painter Wassily Kandinsky was also ruled by synesthesia throughout his life. Kandinsky literally saw colors when he heard music, and heard music when he painted. Kandinsky said that when observing colors, all the senses (taste, sound, touch, and smell) are experienced together. Kandinsky believed abstract painting was the best way to replicate the melodic, spiritual, and poetic power found in music. He spent his career applying the symphonic principles of music to the arrangemen<sup>t</sup> of color notes and chords [77].

The art philosopher Nikolai Hartmann, in his book *Aesthetics* (1953), considered auditory–visual–touch synesthesia in art. Taggart et al. [78] found thar synesthesia is seven times more common among artists, novelists, poets, and creative people. Artists often connect unconnected realms and blend the power of metaphors with reality. Synesthetic metaphors are linguistic expressions in which a term belonging to a sensory domain is extended to name a state or event belonging to a different perceptual domain. The origin of synesthetic experience can be found in painting, poetry, and music (visual, literary, musical). Synesthesia appears in all forms of art and provides a multisensory form of knowledge and communication. It is not subordinated but can expand the aesthetic through science and technology. Science and technology could thus function as a true multidisciplinary fusion project that expands the practical possibilities of theory through art. Synesthesia is divided into strong synesthesia and weak synesthesia [78].

Martino et al. [79] reviewed the effects of synesthesia and differentiated between strong and weak synesthesia. Strong synesthesia is characterized by a vivid image in one sensory modality in response to the stimulation of another sense. Weak synesthesia, on the other hand, is characterized by cross-sensory correspondences expressed through language or by perceptual similarities or interactions. Weak synesthesia is common, easily identified, remembered, and can be manifested by learning. Therefore, weak synesthesia could be a new educational method using multisensory techniques. Since synesthetic experience is the result of unified sense of mind, all experiences are synesthetic to some extent. The most prevalent form of synesthesia is the conversion of sound into color. In art, synesthesia and metaphor are combined [79].

To some extent, all forms of art are co-sensory. Through art, the co-sensory experience becomes communicative. The origin of co-sensory experience can be found in painting, poetry, and music (visual, literary, musical) [80].

Today, the ultimate synesthetic art form is cinema. Regarding the senses, Marks [81] wrote: In a movie, sight (or tactile vision) can be tactile. It is "like touching a movie with the eye", and further, "the eye itself functions like a tactile organ".

Colors can be expressed as embossed tactile patterns for recognition by finger touches, and incorporated temperature, texture, and smell to provide a rich art experience to person with visual impairments. *The Black Book of Colors* by Cottin [82] describes the experience of a fictional blind child named Thomas, who describes color through association with certain elements in his environment. This book highlights the fact that blind people can gain experience through multisensory interactions: "Thomas loves all colors because he can hear, touch, and taste them." An accompanying audio explanation provides a complementary way to explore the overall color composition of an artwork. When tactile patterns are used for color transmission, the image can be comprehensively grasped by delivering graphic patterns, painted image patterns, and color patterns simultaneously [83–85].

This suggests to us the possibility and the justification for developing a new way of appreciating works in which the colors used in works are subjectively explored through the non-visual senses. In other words, it can be inferred that certain senses will be perceived as being correlated with certain colors and concepts through unconscious associations constructed with the concepts. The following works were designed to prove this assumption and to materialize it as a system for multisensory appreciation of artworks for the visually impaired.

An experiment comparing the cognitive capacity for color codes named ColorPictogram, ColorSound, ColorTemp, ColorScent, and ColorVibrotactile found that users could intuitively recognize 24 chromatic and five achromatic colors with tactile pictogram codes [86], 18 chromatic and five achromatic colors with sound codes [87], six colors with temperature codes [88], five chromatic and two achromatic colors with scent codes [89], and 10 chromatic and three achromatic colors with vibration codes [90].

For example, Cho et al. [86] presented a tactile color pictogram system to communicate the color information of visual artworks. The tactile color pictogram [86] uses the shape of sky, earth, and people derived from thoughts of heaven, earth, and people as a metaphor. Colors can thus be recognized easily and intuitively by touching the different patterns. What the art teacher wanted to do most with her blind students was to have them imagine colors using a variety of senses—touch, scent, music, poetry, or literature.

Cho et al. [87] expresses color using part of Vivaldi's *Four Seasons* with different musical instruments, intensity, and pitch of sound to express hue, color lightness, and saturation. The overall color composition of Van Gogh's "The Starry Night" was expressed as a single piece of music that accounted for color using the tone, key, tempo, and pitch of the instruments. Bartolome et al. [88] expresses color and depth (advancing and retreating) as temperature in Marc Rosco's work using a thermoelectric Peltier element and a control board. It also incorporates sound. For example, tapping on yellow twice produces a yellow-like sound expressed by a trumpet. Lee et al. [89] applied orange, menthol, and pine to recognize orange, blue, and green as fragrances.
