**4. Multisensory Integration**

Multisensory (or multimodal) integration is an essential part of information processing by which various forms of sensory information, such as sight, hearing, touch, and proprioception (also called kinesthesia, the sense of self-movement and body position), are combined into a single experience [125,126].

Information is typically integrated across sensory modalities when the sensory inputs share certain common features. Cross-modality refers to the interaction between two different sensory channels. Cross-modal correspondence is defined as the surprising associations that people experience between seemingly unrelated features, attributes, or dimensions of experience in different sensory modalities. Although many studies have been conducted on the cross sensation between the sight and other senses, there are not many studies on the cross sensation between the non-visual senses.

Through the investigation of weak synesthesia that is perceived at the same time by intersecting various senses such as hearing, touch, smell, etc., it explores other sensory information that can be connected with specific form and color information in sight. In this task so far, efforts have been made to create different single-mode sensory perceptions for color. A mapping technology that can be easily recognized by expressing colors as temperature, sound, and scent was designed and tested. However, while this helps visually impaired users perceive and understand colors in completely different ways, it becomes necessary to integrate all perceptual sensations into a single multi-sensory system, where all the senses are perfectly connected and interchanged.

Making art accessible to the visually impaired requires the ability to convey explicit and implicit visual images through non-visual forms. It argues that a multi-sensory system is needed to successfully convey artistic images. It also designed a poetic text audio guideline that blends sounds and effects called "sound painting" to translate the work into an ambiguous artistic sound text [127].

Testing was conducted to extend the results from previous work into a complete multisensory color experience system. The relationships between color–temperature/color– sound have been studied through existing research, and through this, visually impaired people feel or perceive color through temperature or sound. However, the connection between temperature and sound is unclear. When appreciating works using both temperature and sound, we need to explore whether temperature and sound interfere with each other, causing confusion in color perception, or synergize with each other to positively affect color perception.

### *4.1. Temperature and Sound*

Wang et al. [128] explored the putative existence of cross-modal correspondences between sound attributes and beverage temperature. An online pre-study was conducted first to determine whether people would associate the auditory parameters of pitch and tempo with different imagined beverage temperatures. The same melody was manipulated to create a matrix of 25 variants with five different levels of both pitch and tempo. The participants were instructed to imagine consuming hot, room-temperature, or cold water and then to choose the melody that best matched their imagined drinking experience. The results revealed that imagining drinking cold water was associated with significantly higher pitches than drinking both room-temperature and hot water and with a significantly faster tempo than drinking room-temperature water. Next, the online study was replicated with participants in a lab tasting samples of hot, room-temperature, and cold water while

choosing a melody that best matched their actual drinking experience. Those results confirmed that, compared with room-temperature and hot water, the experience of drinking cold water was associated with significantly higher pitches and a faster tempo [128]. One potential explanation for those results is emotional associations [129]. Evidence already exists for cross-modal correspondences between sound and smell [130] and between sound and taste [131] that are mediated by emotion, and both fast tempo and high pitch [132] are associated with increased arousal. The experience of drinking cold water might therefore be associated with a fast tempo and high pitches because it is deemed arousing and refreshing. Hot water, on the other hand, could be associated with soothing, calming warm beverages such as tea. This was especially true in the main study, where the hot water was served at 45 ◦C, a comfortable drinking temperature. It would be interesting to ask participants in an online study to associate pitch and tempo with both extremely hot water (around boiling, at 100 ◦C) and a comfortable 45 ◦C. One might expect the very hot (hence arousing) water to be associated with a faster tempo and higher pitches than the comfortably warm water. Of course, to truly verify the emotional association hypothesis, a future study would need gather information about the emotions that participants associate with each beverage sample [128].

Brunstrom et al. [133] explored oral temperature (e.g., of a beverage), a multi-sensory structure that includes odor and sound in addition to tactile and oral sensations. Successful mappings between temperature and color and then sound and color have been designed and tested. However, although such mapping does help the visually impaired to appreciate and understand colors in a new way, it is important to integrate those perceptual mappings into single multisensory system in which all those perceptual sensations can be perfectly linked and interchanged. In other words, a system is needed to enable the interactions of color, temperature, and sound to work together and give both sighted and visually impaired users the chance to experience "colors" through different perceptual sensations, thereby expanding their experience of colors and what they involve [133].

Cho et al. [87] investigated what color-directed sound have. Melodies with different pitch, tone, velocity, and tempo can be used as color sound codes to easily express the color lightness level. It is necessary to explore the cross-mode relationship between sound and temperature in order to express the color more concisely and perceptibly by integrating sound and temperature simultaneously. Two different coding color schemes with combining temperature and sound are suggested, for example:

(1) Celsius temperature represents six colors (e.g., 38—red, 34—orange, 30—yellow, 26—purple, 22—green, 14—blue), and three sound codes with different pitch and tempo (Table 4) represent three levels of color lightness.

(2) Isaac Newton's RYB color model consists of red, orange, yellow, green, blue, violet, red-orange (warmer red), red-violet (cooler red), yellow-orange (warmer yellow), yellow-green (cooler yellow), blue-violet (warmer blue), and blue-green (cooler blue). The color hue is expressed as sound (like in Table 4) and the warm and cold colors as two temperatures (e.g., 34 and 22 degrees Celsius). High color lightness (light), medium color lightness (muted) and low color lightness (dark) can be coded with temperatures like 14, 30, and 38 degrees Celsius, respectively. Combining temperature and sound in this way makes it simpler and easier to identify more colors, including warm/cool colors.

### *4.2. Temperature and Scent*

Wnuk et al. [134] investigated the bases of those cross-modal associations, suggesting several possibilities, including universal forces (e.g., perception), and culture-specific forces (e.g., language and cultural beliefs). They examined odor–temperature associations in three cultures—Maniq, Thai, and Dutch—that differ with respect to their cultural preoccupation with odors, their odor lexicons, and their beliefs about the relationship between odors (and odor objects) and temperature. Their analysis revealed cross-modal associations that could not be explained by language but could be the result of cultural beliefs. Another possibility is that odor–temperature associations do not depend on cultural beliefs but are universal, perhaps due to shared physiology. It is often assumed that odors are associated with hot and cold temperatures because odor processing can trigger thermal sensations, such as the connection between coolness and mint. They found that menthol (peppermint) and cineole (eucalyptus) were consistently matched with the temperature term "cool". Laska et al. [135] also found that menthol (peppermint) and cineol (eucalyptus) consistently match the temperature conditions (cooling) [134].

Madzharov et al. [136] pretested six essential oils, three of which we expected to be perceived as warm scents (warm vanilla sugar, cinnamon pumpkin, and spice) and three as cool scents (eucalyptus–spearmint, peppermint, and winter wonderland). Following an established procedure (see Krishna, Elder, and Caldara 2010), 33 participants evaluated each scent on perceived temperature and liking ("smells like a cool/warm scent", seven-point scales). Of the six scents, cinnamon and vanilla were rated as the warmest, and peppermint was rated as the coolest. Cinnamon and peppermint were significantly different on the temperature dimension, as were vanilla and peppermint. According to Mackenzie [137], cinnamon and vanilla not only taste good to many people, but the scent of cinnamon or vanilla can invoke a warm, comforting feeling [136].

As shown in Table 5, orange and chocolate were used to easily express the color lightness level, and chocolate and menthol to express the temperature "warm/cool", respectively. It is necessary to explore the cross-mode relationships between scent and temperature, and scent and color lightness to express the color in terms of warm/cool and light/muted/dark. The following color-coding scheme with integrating temperature and scent is suggested.

Celsius (◦C) temperature represents six colors (e.g., 38—red, 34—orange, 30—yellow, 26—purple, 22—green, 14—blue), and scents like orange and pine can convey two levels of color lightness (light/dark) (Table 5). Finally, warm and cold colors can be expressed by scents like chocolate and menthol.

Note that the six color hues cannot be expressed as scent since only four colors are associated with scent, as shown in Table 5. Combining temperature and scent in this way makes it simpler and easier to convey more colors, including warm/cool colors.

### *4.3. Scent and Sound*

Researchers have started to document the existence of cross-modal correspondences between olfactory and auditory stimuli. For instance, Belkin [138] and Piesse [139] showed that people matched a series of different odors with sounds that differed in pitch. Piesse [139] introduces the idea that Olfaction can be described in ways that correlate to the musical notes on a diatonic scale. Those results were extended by [140,141], who found that people tended to match certain odors with the timbres of musical instruments.

Crisinel et al. [141] found that odors were preferentially matched to musical features: for example, the odors of candied orange and iris flower were matched to significantly higher pitches than the odors of musk and roasted coffee. Meanwhile, the odor of crème brûlée was associated with a more rounded shape than the musk odor. Moreover, by simultaneously testing cross-modal correspondences between olfactory stimuli and matches in two other modalities, they were able to compare the ratings associated with each correspondence. Stimuli judged as happier, more pleasant, and sweeter tended to be associated to both higher pitch and a more rounded shape, whereas other ratings seemed to be more specifically correlated with the choice of either pitch or shape. Odors rated as more arousing tended to be associated with the angular shape, but not with a particular pitch; odors judged as brighter were associated with higher pitch and, to a lesser extent, rounder shapes [141]. The emotional (hedonic) similarity between olfactory and auditory information could be crucial to both cross-modal correspondences and multisensory information processing [142].

Currently, Touch the Sound [143] and Perfumery Organ [144] are cross-sensory media works, but research on color expression is extremely rare. In Perfumery Organ [144], the fragrance is scented when played on the piano using the "incense" that connects the

fragrance and sound devised by the perfumer Septimus Piesse [139], who matches "do (C4)" with rose, "le (C4)" with violet, and "mi (C4)" with acacia.

In Table 5, orange and chocolate can be used to easily express the color lightness level, and chocolate and menthol to express the temperature "warm/cool", respectively. It is necessary to explore the cross-modal relationships between scent and sound, and between scent and color lightness to express the color in terms of "warm/cool" and "light/muted/dark" and make it more easily perceptible. For example, the following color-coding scheme with integrating scent and sound is suggested.

Six colors can be expressed with sounds (Table 4). Scents of orange and pine convey two levels of brightness (brightness/darkness) (Table 5). Finally, scents like chocolate and menthol convey warm and cold colors (Table 5). Combining scent and sound in this way can make it simpler and easier to convey more colors, including warm/cool colors.

### *4.4. Scent and Shape*

Odors rated as more arousing tended to be associated with the angular shape, but not with a particular pitch; odors judged as brighter were associated with higher pitch and, to a lesser extent, rounder shapes [140].

Humans do not arbitrarily attach sounds to shapes, as can be seen in the Kiki/Bouba effect [145,146]. Köhler [145] found that 95–98% assigned the name "bouba" to the rounded shape and "kiki" to the jagged shape, Figure 6.

**Figure 6.** (**Left**) kiki: angular, jagged shapes; (**Right**) bouba: smooth, rounded shapes [146].

Adeli et al. [147] investigated the cross-modal correspondences between musical timbre and shapes. One hundred and nineteen subjects (31 females and 88 males) participated in the online experiment. Subjects included 36 claimed professional musicians, 47 claimed amateur musicians, and 36 claimed non-musicians. Thirty-one subjects have also claimed to have synesthesia-like experiences. Subjects have strongly associated soft timbres with blue, green or light gray rounded shapes, harsh timbres with red, yellow or dark gray sharp angular shapes. This is consistent with Kiki–Bouba experiment where subjects mostly chose a jagged shape for Kiki and a rounded shape for Bouba [145–148]. It is also consistent with Parise's findings [148] where subjects associated sine waves (soft sounds) with a rounded shape and square waves with a sharp angular shape [147].

Hanson-Vaux et al. [149] investigated how children relate emotions to smells and 3D shapes. Fourteen participants (ages 10–17 years) performed a cross-modal association task that gave emotional character to the transformation of the "kiki"/"bouba" stimulus presented as a 3D type model with lemon and vanilla flavors. The results of the study confirmed the association between the combination of the angular shape ("kiki") and the stimulating lemon scent, the round shape ("bouba") and the soothing vanilla scent. This expands the new results for the cross-mode response in terms of stimuli (3D rather than 2D shapes), samples (children), and delivered content compared to previous studies. We explored how these findings could contribute to the design of more comprehensive interactive multi-sensory technology.

Metatla et al. [150] investigated cross-modal associations between 20 odors (a selection of those commonly found in wine) and visual shape stimuli in a sample of 25 participants (mean age of 21 years). Two of the odors were found to be significantly associated with an angular shape (lemon and pepper) and two others with a rounded shape (raspberry and vanilla). Principal component analysis indicated that the hedonic value and intensity of odors are important in this cross-modality association, with more unpleasant and intense smells associated with more angular forms.

Lee et al. [89] investigated cross-modal associations between scents and visual shape stimuli like "kiki" and "bouba". The participants of the experiment were visually presented at the same time a paper with an angular shape and a rounded shape that corresponds to the words "kiki" and "bouba", respectively. The results of the study (Table 5) confirmed the association of the angular shape ("kiki") with the stimulating menthol, pine, and orange scents (associated with blue, green, and orange colors), and the round shape ("bouba") with the soothing chocolate scent (associated with brown). There was no significant difference in sharpness between menthol, pine and orange.

In summary, red and yellow are associated with "kiki" and blue and green with "bouba" [149]. Lemon is associated with an angular shape and vanilla with a rounded shape [149,150]. From [147–150], we can conjecture red and yellow are associated with lemon scent, and blue and green are associated with vanilla. From [89], orange, menthol, and pine scents correspond to orange, blue, and green that are associated with an angular shape. Additionally, chocolate scent corresponds to brown, which is associated with a rounded shape. Therefore, the results of research on the relationship between fragrance and shape might differ according to the cultural background of the participants.
