*2.1. Tactile Graphics*

Tactile graphics (TG) are made using raised lines and textures to convey drawings and images by touch. They are frequently used by blind and visually impaired people because the tactile modality is the best for their graphical image comprehension [15]. Their use is recommended where spatial relationships among the graph's objects are important [16], such as simple graphs, diagrams, and drawings. Unfortunately, they are ineffective to express visual information of complex images [17,18], such as those present in many visual artworks. For this case, adding Braille labels is of limited use due to the large space needed by the Braille characters to be legible. Moreover, including labels within the artwork area obstructs exploration. Advances in low-cost prototyping and 3D printing technologies bring the potential to tackle the complexity of expressing complex images without exploration obstruction by adding interactivity to tactile graphics.

### *2.2. Interactive Tactile Graphics and 3D Models*

In the last decades, researchers have explored the improvement of tactile graphics accessibility by adding interactivity through diverse technologies. Some of the improvements are better content exploration [18], learning facilitation [19], and expansion of the amount of information provided without over-complications [20]. Table A1 summarizes several of these projects and their interaction technologies. Three early works are NOMAD [21], The Talking Tablet [22], and IVEO [23], all of which function by placing a tactile graphic on

a high-resolution touch-sensitive pad that detects user touch gestures that trigger audio descriptions. This method provides independent and detailed access to graphic elements, and since it does not rely on Braille, the possible audience is broader. Taylor et al. [24] and LucentMaps [25] make use of the touch screens in mobile devices to detect user–touch interactions in a portable way. They attach 3D printed tactile overlays of city maps to the device screen. Taylor et al. [24] 3D print sections of the overlay using conductive filament to provide interaction points on discrete sections of the map. LucentMaps instead uses translucent filament for their overlays coupled with a mobile application that visually highlights sections of the overlay using the device screen. MapSense [26] also uses a touchscreen to identify user touch gestures and conductive tangible tokens placed on the surface. The tangibles are additionally infused with smell and taste to foster reflective learning and memorization. Using touch-sensitive surfaces to detect user input and trigger audio feedback increases the amount of information communicated to the user. However, this approach is limited to thin overlays. Otherwise, the system can't recognize the touch gestures.

An alternative approach is using cameras to track either the content or the user's hands. CamIO [27], Tactile Graphics with a Voice [28,29], and The Tactile Graphics Helper [30] are examples of projects using this approach. The Tactile Graphics with Voice projects work by using a mobile or wearable device's camera to identify QR codes printed along a tactile graphic. Then, the system tracks the user's hand to trigger localized verbal descriptions. CamIO and The Tactile Graphics Helper use mounted cameras that identify the content using image processing algorithms, instead of using QR codes or visual markers. With the exception of CamIO, the previous projects focus on adding interactivity to 2D tactile graphics, and mainly propose their use for STEM (science, technology, engineering and mathematics) education and orientation and mobility improvement. Both approaches are effective to improve the amount of information and the comprehension of the spatial arrangemen<sup>t</sup> of images. However, to facilitate comprehension, they abstract the complexity of images to contour lines, which hinders the aesthetic aspect and exploration of artwork images.

3D printing opens up the possibility to create low-cost reliefs and 3D models of objects with added expressive volume. Holloway et al. [31] propose a touch interactive prototype that uses 3D printed volumetric representations of map models embedded with discrete capacitive touch points that users can touch to trigger audio descriptions. This approach improved the short term recollection and the understanding of the relative height among the map elements. Other studies focused on symbolic representation on 3D maps models, like Holloway et al. [32] and Gual et al. [33,34], they report improvements in terms of accuracy, efficiency, and memorability compared to two-dimensional symbols. Alternative methods to add interactivity involve using other type of devices. For example, pen-shaped devices like The Talking Tactile Pen [35] or wearables like the ring-shaped Tooteko [36]. In this approach, the user must hold or wear the device, which can detect sensors embedded in the tactile graphic or models on approximation.

#### *2.3. Interactive Multimodal Guides for Blind and Visually Impaired People*

The body of work on interactive multimodal guides focused on artwork exploration is limited, as seen in Table A1. However, there are several related works. The American Foundation for the Blind offers guidelines and resources for the use of tactile graphics for the specific case of artworks [37]. Cho et al. [38] present a novel tactile color pictogram system to communicate the color information of visual artworks. Volpe et al. [39] explore the semi-automatic generation of 3D models from digital images of paintings, and classifies four classes of 3D models (tactile outline, textured tactile, flat-layered bas-relief, and bas-relief) for visual artwork representation. After an evaluation with fourteen blind participants, the results indicate that audio guides are still required to make the models understandable. Holloway et al. [14] evaluated three techniques for visual artwork representation: tactile graphic, 3D print (sculpture model), and laser cut. Notably, 3D print and laser cut are

preferred by most participants to explore visual artworks. Hinton [40] describes the use of tactile graphics of visual artworks made using thermoforming intended to be explored along with tape recordings. Blind study participants reported that the approach helped them understand the space and perspective of the artworks and found the approach fun, interesting, informative, and even stimulating to their creative efforts.

There are the few projects that add interactivity to visual artwork representations and museum objects. Anagnostakis et al. [41] use proximity and touch sensors to provide audio guidance through a mobile device of museum exhibits. Vaz et al. [42] developed an accessible geological sample exhibitor that reproduces audio descriptions of the samples when picked up. The on-site use evaluation revealed that blind and visually impaired people felt more motivated and improved their mental conceptualization. Leporini et al. [43] explore the use of a three-dimensional archeological map and fascade models to communicate historical, practical, and architectural information on demand, using 3D printed buttons with success to provide autonomous and satisfying exploration. Reichinger et al. [44–46] introduce the concept of a gesture-controlled interactive audio guide for visual artworks that uses depth-sensing cameras to sense the location and gestures of the user's hands during tactile exploration of a bas-relief artwork model. The guide provides location-dependent audio descriptions based on the user's hand position and gestures.

We designed and implemented an interactive multimodal guide prototype based on the needs found through our preliminary study described in Section 3.1 and inspired mainly in the related works Holloway et al. [31] and Reichinger et al. [44]. Table 1 compares the main technical differences between the related works and our approach. Besides these differences, this work introduces a comparison between our approach and using traditional tactile graphics to measure potential improvements of the multimodal approach.


**Table 1.** Features of the proposed interactive multimodal guide and selected related works.

\* This work.

### **3. Materials and Methods**

### *3.1. Formative Study*

To better understand the current state of the accessibility tools available to experience visual artworks and to explore the requirements for the use of interactive multimodal guides, we conducted a formative study with blind and visually impaired participants, art museums and gallery staff, and artists.

3.1.1. Accessible Visual Artworks for Blind and Visually Impaired People

The formative study focused on the current access to visual artworks through tactile graphics and other means with eight blind and visually impaired participants, with an average age of 29.13 (standard deviation of 7.7). Other characteristics of the participants are described in Table 2. Of the eight participants in the study, three (37.5%) are male, and five (62.5%) are female. While five (62.50%) of the participants attend university studies, three (37.5%) of them work. All the participants gave signed informed consent based on the procedures approved by the Sungkyunkwan University Institutional Review Board.


**Table 2.** Characteristics of blind and visually impaired participants in our formative study.

We followed a semi-structured interview focused on the access and availability of tactile materials at museums, galleries, and through their education. Moreover, we inquired about their experience when using tactile graphics and interactive guides, if any. While all the participants stated having experience using tactile graphics, most of the encounters with this type of materials were limited to educational materials and tactile books during their early education or related to STEM subjects and maps. Four participants stated having experience with tactile graphics related to visual artworks. All the participants that said having experience with tactile graphics in the art fields had access to them during their primary and secondary studies. Only two mentioned having experienced them during a visit to a museum or gallery. All of the participants expressed having visited a museum or art gallery; they reported that the most common accessible tools during their visit were guided tours and the use of audio guides. Seven of the participants mentioned that they were accompanied by someone (relatives or friends) during their visits. They added that they mostly relied on that person's comments and help to use the audio guide during their visit to experience the artworks.

Regarding their experience exploring tactile graphics, the participants mentioned that they are convenient to understand simple diagrams of mathematical concepts or simple graphics in educational fields, learning language characters, and storybooks. Mixed results were reported in their use for tactile maps. Three participants considered tactile graphics easy to understand, while five found them over-complicated or not very useful. However, all of the participants with previous experience with tactile graphics of visual artworks stated dissatisfaction due to their limitations. In particular, one participant commented: *"FP2: Using the tactile graphics is a hit and miss. If the contents are simple and separated is easy to get an idea of what the picture looks like, but often there are so many shapes and textures that is difficult to imagine what the picture looks like, it becomes hard, like thinking about math, art is not supposed to be like that."* This reflects the known problem of producing tactile graphics of complex images, which is usually dealt with by simplifying and abstracting the objects in the image. However, this approach often doesn't solve the problem in the case of tactile artworks. *"FP3: So much detail is lost when touching a tactile graphic. Even if I can find and feel the silhouette of a person or their face, I cannot know if the person in the painting is smiling or crying, and that's what people usually talk about."* Another problem is the challenge to represent perspective and volume. *"FP3: When exploring a tactile graphic everything is on the same level, there's no depth like in the real world. If it's a landscape, I don't know what is in front*

*and what's on the back. Even something simple like a ball, I only feel a circle, and many things can be a circle. I'm told that in the painting you can know it's a ball because of the color and shadows, but I just feel a circle."* Despite the shortcomings, the participants expressed the need for tactile graphics and desired for them to be available for more artworks and more locations. *"FP5: Even when they are not perfect (tactile graphics), they are still useful to know what is where in the painting, I still can be in the conversation. I just hope they were available in more places and for all the works."*

#### 3.1.2. Accessible Visual Artwork at Art Museums and Galleries

Some of the participants in the formative study mentioned the shortage in the availability of tactile graphics or other accessibility tools in their visits to art museums and galleries. We met with a couple of administrators and curators at a national art museum, a private art gallery, and an accessible gallery at a social welfare center for blind and visually impaired people, to shed some light on their approach and efforts towards the accessibility to their collection. At the national art museum, they described several of their initiatives towards accessibility. Their current effort is mostly directed to accessible tours. Besides the tours and available audio guides, some of their exhibitions are made accessible through 3D-printed models that can be explored by touch. However, this tool is not always available, and it is used mostly for large modern art installations. The private gallery just offered guided tours by its staff. There were two main concerns. First, any accessible tool or display must be unobtrusive. One of the concerns was that any display co-located with the artwork can become a distraction and deviate the attention from the artwork. The second concern is about the contents. The administrators commented that presenting the artwork through a different medium than the one used by the artist could have implications in the message and intention that the artist wanted to express. Because of this, the use of accessible exhibits is more often available for modern artworks, where the artist can provide guidelines or collaborate in the development of the exhibits or even make their artworks considering accessibility needs.

### 3.1.3. Accessible Visual Artwork and Artists

We interviewed two artists separately to inquire about the use of accessibility tools and other mediums to experience their art. To generate richer insights, we provided one tactile graphic representation of a painting and discussed it with them. Both artists agreed on the importance of making visual art more accessible to blind and visually impaired people and that it may require the introduction of other tools or mediums. To this end, they strongly suggested collaboration with the author or experts when possible, noting that while the artist may not be an expert on the added medium, it can provide feedback to improve it. One of the artists expressed his concern regarding tactile graphics *"Artist 1: I believe too much emphasis is placed on what is in the painting and not the painting itself. Yes, the recognition of shapes, objects, colors, and elements is relevant, but I dare to say it is not the most important aspect. Viewers should not be passive, just saying to them 'this is this' or 'this means this' is a failure. The goal of my art is to cause a reaction when someone sees it, they (viewers) should think, they should react. That's what experiencing art is."* We believe that this is a very relevant point, since most of the research literature is centered in the improvement of recognition of the objects in the painting, but there is almost no improvement related to the reaction and interpretation studies when using accessible artwork guides.
