Mobile AR: User Evaluation in a Cultural Heritage Context

Abstract

The growing number of mobile augmented reality applications has been favoring its awareness and usage among diversified areas. Focusing on cultural heritage applications, this study presents an evaluation of a mobile augmented reality application tested at Conimbriga, an archaeological site. The prototype developed for this purpose, named DinofelisAR, allowed users to view, over 360 degrees, a majestic reconstruction of a Forum from the Roman Era superimposed over its current ruins. Thus, users were able to keep perceiving the present-day surroundings of a Roman city in ruins while, at the same time, had the possibility to explore the matching virtual model. The results presented, arising from 90 participants involved in this evaluation, praise the sense of opportunity for new augmented reality solutions targeted at cultural heritage sites.

1. Introduction

Cultural Heritage (CH) places, such as museums or archaeological sites, envisage pleasant visits for their visitors, while visitors want to learn, enrich themselves and have fun. This is evidenced in an internal survey made in the Conimbriga Monographic Museum-National Museum in 2014, where the main reason identified as the reason that motivated visitors to come to Conimbriga was cultural enrichment. Other reasons stated in this same survey are related to leisure time, knowledge improvement about history and archeology, to ruins’ mosaics, to architecture and urbanism, and to the object collection inside the museum.

To indulge the visitors’ expectations, museums are always seeking for different and innovative solutions to provide information to their guests. Aside from their historic artefacts and infrastructures (or what remain of them), informative images are frequently added to specific places (digital and real), mock-ups reconstructions are showed, multimedia content (audio-visual information) is displayed, etc. Therefore, new technological approaches, such as augmented reality, has been exploited in the last few years among different museums. To better understand how this technology would be perceived by people, a proper and user-oriented use of technology should be prospected to CH contexts, attempting to come out with reliable solutions to profit visitors and institutions.

The use of Augmented Reality (AR) has a positive impact on users’ perspective and has been recognized as a good solution to be used in diversified areas [1]. However, AR still has not gained a solid presence in our digital society so far, expressing AR as a present and future topic to work on. Although this technology has seen an increasing popularity grow among a vast variety of areas, such as publicity [2], entertainment [3], education [4,5], medicine [6], architecture [7], manufacturing [8], and CH [9,10,11], which is the particular case for this study. Although being stated as a promising tool according to literature, producing AR experiences available for visitors in cultural spaces whereby they can experience it during their visits is not a common feature available yet—when compared to the above-mentioned methods of images, flyers, or mock-up reconstructions. This gap is even more noticeable in AR experiences for outdoor spaces, such as archaeological sites, a fact that motivated this research. It is proved that AR applications in CH sites can add additional value to users’ experiences [12]. However, due to some repercussions and limitations found in AR literature, before becoming broadly commercialized in tourism contexts, AR research in CH still has a long way to go [13].

Notwithstanding resourceful museums, which are already facilitating the opportunity for users to use AR in their spaces, sparking the interest of new users is important. The cases of “Augmenting the Historic City: Trade and Merchant’s Life in Ribe”, in Sydvestjyske Museum (Denmark) [14], “The Skin and Bones” application, in exhibit at the Smithsonian in Washington (USA) [15], and the project “England’s Historic Cities”, available for 12 museums in England [16] are worth mentioning.

This study aimed to evaluate a mobile augmented reality application created for outdoors to represent an ancient building—arising from the Roman civilization—which is currently in ruins. This application was tested by visitors to the Conimbriga Ruins, managed by the Conimbriga Monographic Museum-National Museum, Portugal.

Taking into consideration the target audience and the place where it is intended to evaluate an AR solution, some thoughts and requirements were traced: there is no Internet or electricity available on site; outstanding marks should be avoided to not hamper a regular visit to the site; devices that would need maintenance such as batteries should be avoided; weather conditions cannot be controlled; and the users are non-experts and may not understand the limitations that the app may have.

The current study intended to understand users’ feedback when handling a mobile AR app by themselves. The visitors who accepted the challenge used a smartphone to experience the AR app—DinofelisAR—freely, with no time limit. The findings obtained with this experiment were based on the results acquired from questionnaires, and allowed to trace valuable insights about the use and the performance of an AR app in cultural heritage sites.

2. Methods

A prototype was developed, tested, and evaluated in-situ by end-users. The following sections explain the procedure to accomplish the results to understand how mobile AR technology was perceived among visitors of Conimbriga Ruins.

2.1. Prototype Development

DinofelisAR prototype for Android devices was implemented in Unity 3D with Vuforia SDK, having an image as mark for the experience. However, given the variety of techniques and frameworks available to develop augmented reality solutions presently, DinofelisAR prototype was outlined relying on previous studies focused on identifying the more suitable techniques and frameworks to develop an AR system targeted at being used in an archaeological site, keeping in mind that user tests would be performed in loco [17,18].

2.1.1. AR (Augmented Reality) Tracking Techniques

Depending on the context where the AR system will be used and on the people who will use it, the tracking technique should be selected accordingly. The literature presents techniques such as the mobile localization techniques proposed by Bae et al. ([19]): (1) sensor-based localization; (2) marker-based localization; (3) visual simultaneous localization and mapping (visual SLAM); and (4) model-based localization. Another classification was presented in 2015: (1) magnetic tracking; (2) vision based tracking; (3) inertial tracking; (4) GPS tracking; and (5) hybrid tracking [20]. More recently, considering the state of the art of near-eye display and tracking technologies for virtual and augmented reality, the authors classified tracking technologies as: (1) mechanical (e.g., physical linkage as the first HMD for AR invented by Sutherland in 1968); (2) electromagnetic (e.g., magnetic sensing); (3) inertial (e.g., accelerometers and MEMs); (4) acoustic (e.g., ultrasonic); (5) optical (e.g., computer vision); and (6) hybrid (combination of technologies) [21].

For the current study, which evaluated an AR application to be used at an archaeological site, some prototypes exploring a vision based tracking were tested. To do so, 2D images and 3D models were tested to be used as marks or “targets”, as referred by Vuforia. A first prototype was developed by scanning real rocks, in order to use them as markers to trigger and track the user while experiencing AR. Figure 1 presents two of the rocks used for this prototype.

The tests made with this approach revealed great sensitivity to any changes in lighting, especially, when objects were facing the sun, which resealed to be a critical remark for the object recognition outdoors [17].

Thus, 2D images were tested for the following prototypes. Two types of images were examined: computerized images and captured images (from real world). Two examples may be observed in Figure 2.

According to a previous study, the computerized images provided more stable results regarding to steadiness and to positioning accuracy [18].

2.1.2. AR SDKs

A comparative study was carried out to understand which SDKs would fit the needs of DinofelisAR, considering the requirements of this app [18]. This study compared a long list of SDKs considering their characteristics and features. From this comparison, some prototypes were implemented based on different SDKs (such as Wikitude SDK, Vuforia SDK, and Kudan SDK) to evaluate each app aiming to choose the most suitable one to test in Conimbriga Ruins. It was suggested that Vuforia SDK, due to its good performance and its feature “Extended Tracking” for tracking without the image target within the camera’s field of view, would better fit the specific requirements for the current study.

Thus, DinofelisAR was implemented with Vuforia SDK, having a computerized image as mark for the recognition and tracking, and using the Extended Tracking feature, to allow the tracking based in natural features, keeping the experience possible even when the image target is out of the camera field of view.

2.2. Prototype Testing In-Situ

A mobile AR prototype, DinofelisAR, was designed to allow the visitors of an archaeological site to experience mobile AR technology in-situ.

2.2.1. Pilot-Tests

The pilot-tests performed with DinofelisAR took into consideration a prediction of visitors’ profiles who could be part of this experience in-situ, based on statistics collected about Conimbriga’s visitors [22]. This analysis led us to select four profiles to carry out the pilot-tests: a foreign couple in theirs 50s, two 16-year-old Portuguese girls, and a Portuguese–French girl in her 30s, who understands Portuguese but it is not her native language.

For performing the pilot-tests, the researcher presented the app as if they were final users and the volunteers launched and experienced the DinofelisAR app themselves. An estimation time of the experiences was achieved: on average, 1 min 32 s for each experience. After the experience, they were asked to fill-in the questionnaire developed accordingly, a procedure that took them approximately, 2 min to finish. At the end, they were asked to provide their opinions and insights about the experience.

This process allowed us to improve our experience, performing some adjustments related to the instructions given to the participant before they tested the app and to clarify some terms in the questionnaires that were misinterpreted by the pilot users.

2.2.2. Procedure for Users’ Experiment

Aiming to understand users’ feelings regarding the usage of a mobile AR app when visiting an archaeological site, the main purpose of this evaluation was to test the mobile AR app, in-situ, enabling visitors to perceive a virtual reconstruction of the ancient Forum superimposed on its ruins using a mobile device. Aiming to ascertain if the developed app meets certain performance goals, an evaluation was carried out through questionnaires. This paper presents answers given by random Conimbriga’s visitors who embraced the invitation to try the DinofelisAR app. Most people accepted the challenge: 104 people were inquired and only six declined the invitation. There were also eight visitors who tested the app, but did not fill-in the questionnaires (due to lack of time or they were not self-assured with the available questionnaires languages—available in English and Portuguese). Large group visits, such as excursion or tours, were avoided for this experiment due to logistical limitations.

For the AR experience, a smartphone (a Motorola third generation 5.0” was used, it is a quad-core 1.4 GHz Cortex-A53, 1 GB RAM, camera 13 MP, f/2.0) was given to each participant in order to perform the following steps:

• Launch DinofelisAR application.
• Point the smartphone’s camera to the provided image, used as marker for the AR feature, until they detect a virtual cube overlapped on the image, as illustrated in Figure 3;
• Through the smartphone’s camera, explore the surroundings to observe the virtual model added (see Figure 4).
• Fill-in the questionnaire (English or Portuguese version).

3. Results Obtained

In total, 90 visitors participated in this experience by trying the app and filling-in the questionnaire. Consent forms were provided and signed by each participant. More detailed characteristics regarding to the participants who took part to this evaluation can be found in [23].

Before the users’ interest and performance questions, participants were asked about their motivation to be part of this study, where it was found that the majority (72%) of the participants pointed out their interest in having augmented reality applied to archaeological context. Those who affirmed not to know the AR technology stated their curiosity to try it (18%) while 10% argued other reasons [23].

When asked about their level of agreement to the given statements, users were invited to use a Likert scale, choosing between 1 (for strongly disagree) and 5 (for strongly agree).

3.1. Users’ Interest Results

As stated in the previous study related to users’ perspective about a mobile AR application in CH [23], participants highlighted an expressive level of satisfaction when using mobile AR technology in Conimbriga’s Ruins.

The gathered results disclose participants’ agreement that AR contributed to a more dynamic and active visit in the Forum (94%). They also stated that fun was also increased (94%). Figure 5 shows a histogram related to the fun when using the mobile AR app DinofelisAR.

Furthermore, 92% of the participants affirmed to feel more involved when using the AR app in their visit.

Considering the above-mentioned reasons visitors decided to visit Conimbriga, where cultural enrichment was indicated as the main reason for them to visit CH sites ([22]), participants of the current study were asked if AR applications, such as DinofelisAR, can foment cultural enrichment in their visits. The results show a majority agreement of 97%.

As a follow-up to this enrichment, 91% of participants stated that they became able to better describe the Forum characteristics—e.g., size, colors, and architecture (Figure 6).

As Conimbriga’s visitors, they were asked if they believed that the use of this technology could contribute to an increase number of visitors to this archaeological space. Ninety percent of the respondents agreed or strongly agreed.

Regarding the interest of installing this kind of AR mobile applications on their own smartphones (see Figure 7), regardless of the average age of this study sample, participants confirmed their desire to install it having the intention to use it among CH spaces.

Given the variables presented, some correlations were found. A strong correlation was found between fun and the ability to better describe the Forum characteristics—e.g., size, colors, and architecture. This correlation, with a sample of 87 answers, shows a Pearson correlation factor of 0.549 with a significance (two-tailed) of 0.000, highlighting a positive strong correlation statistically significant (

Table 1. Correlation based on Pearson’s correlation between fun and ability to better describe the Forum characteristic.

		Fun	Forum Characteristic
Fun	Pearson Correlation	1	0.549 **
	Sig. (2-tailed)		0.000
	N	87	87
Forum Characteristic	Pearson Correlation	0.549 **	1
	Sig. (2-tailed)	0.000
	N	87	87

** Correlation is significant at the 0.01 level (2-tailed).

).

In addition, a strong statistically significant positive correlation was identified between enrichment contribution of this app and the installation interest expressed by users: a Pearson correlation of 0.557 with a significance (two-tailed) of 0.000 (see

Table 2. Correlation based on Pearson’s correlation between enrichment contribution and installation interest.

		Enrichment Contribution	Installation Interest
Enrichment Contribution	Pearson Correlation	1	0.557 **
	Sig. (2-tailed)		0.000
	N	88	87
Installation Interest	Pearson Correlation	0.557 **	1
	Sig. (2-tailed)	0.000
	N	87	87

** Correlation is significant at the 0.01 level (2-tailed).

).

3.2. Performance Results

Regarding the ease of use, despite the 61% of people declaring that they had never tried this technology before, 96% of the respondents agreed or strongly agreed that AR technology is easy to use (see Figure 8).

Figure 8 shows that, regardless of their previous experience using this technology, most participants stressed AR as being easy to use.

Moreover, the majority (88%) claimed that only a few moments after beginning to use AR technology, they were already comfortable using it and understood how to use it successfully. With a slight difference between female and male respondents, Figure 9 shows the speed of learning how to use AR technology speed.

Some performance issues were considered when choosing the technology used for this prototype. In a previous study where an overview of recent technologies to implement mobile augmented reality systems was made [18], issues related to positioning and errors of positioning were covered. In this study, the framework Vuforia was chosen after some performance tests as it has slightly better results for an augmented reality application running outdoors. Nonetheless, several foreseeable issues were addressed in the questionnaires. The great majority disagreed with the presented issues as we may notice in the following results.

The majority (84%) did not feel lost during the experience, without knowing very well where to point the smartphone while exploring the virtual model.

Regarding to other referred issues—oscillation and errors of positioning—a strong correlation was identified in the current study, as shown in

Table 3. Correlation based on Pearson’s correlation between oscillation issues and errors os positioning when exploring the surroundings.

		Oscillation	Bad Positioning
Oscillation	Pearson Correlation	1	0.344 **
	Sig. (2-tailed)		0.001
	N	88	87
Bad Positioning	Pearson Correlation	0.344 **	1
	Sig. (2-tailed)	0.001
	N	87	87

** Correlation is significant at the 0.01 level (2-tailed).

.

A strong statistically significant positive correlation was found between oscillation issues and errors os positioning.

When questioned about whether an excessive oscillation event was sensed in the virtual representation, users disagreed or strongly disagreed with the allegation (75%), as displayed in Figure 10.

Similarly, when asked about a hampering event related to the positioning of the virtual model while exploring the surroundings, most participants did not consider it as an issue. Nonetheless, 13% agreed with the referred statement (Figure 11).

In addition, participants were asked about the need of being near the sheet of paper (technically, the mark) to use the DinofelisAR. According to the results, this situation restricted the users’ experience by 31%. This issue was reflected in participants’ suggestions: 28% stated that DinofelisAR could be improved by allowing to walk freely in space, while many participants (45%) stated that this situation did not limit their experience.

4. Discussion

The use of mobile AR in CH contexts was first implemented close to the beginning of the millennium, as can be seen in the MARS project [24] and in the ARCHEOGUIDE project [25]. Back then, the user needed to carry a set of heavy devices and these systems were considered mobile. However, the concept of mobile has changed greatly: currently, mobile systems need to be much lighter (e.g., handheld devices or glasses). These forerunner projects were followed by other research solutions related to mobile AR applied to CH sites aiming at exhibition enhancement, reconstruction, and exploration purposes [26]. Previous studies during the last decade, e.g. the use of AR children to imagine certain artifacts and situations related to historical contexts [27], have shown us the perceived enjoyment and usefulness of AR technology in CH contexts [28] as well as enhanced customer value [29]. The presented study provides data that confirm the particular timeliness of mobile AR usage among cultural sites.

The subjective data collected provide indications of a great acceptance of AR apps applied to cultural heritage, regarding to fun and learning (e.g., the ability to better describe the Forum characteristic). These very positive results, where users affirmed they will use the system on their personal smartphones, suggest that the use AR may attract more visitor.

According to the results obtained, the performance issues identified by users did not appear to interfere with their willingness to use mobile AR in CH. However, since the technology used was a novelty for most participants, it may have had the advantage of surprise. For this reason, and because AR technology is becoming increasingly popular, in future implementations, the presented issues must be minimized or fixed, in order to follow the evolution of technology.

Future research directions should provide understandings related to users’ involvement during AR experiences. The current study, as well as previous studies related to AR implementations, did not cover evaluations of the sense of presence when experiencing AR and this lack of information led us to an unaware of how to create more engaging experiences with AR in CH. One way to do it would be the use of this technology to stimulate more human senses, such as audio, touch, and smell. A deep understanding of the users’ intention to use AR technology to accomplish successful approaches should also be considered.

Regarding to technical implementation, future applications should provide better tracking behavior than the extended tracking used for the current study. Considering the willing of visitors to walk around and explore the site when experiencing the AR app, it would allow the user to walk around and explore the virtual elements, instead of staying in one place and just rotating the device 360 degrees. In fact, humans are more used to exploring their surroundings by walking around, getting close to elements we want to see, and not staying at one place and rotating. Other improvements may include investigating the suggestions and issues reported by users, such as finding a solution to avoid the sun on the screen, having a bigger device to better observe the virtual elements, or adding avatars and animations to better understand the size of the augmented elements.

5. Conclusions

This study aimed to implement, test, and evaluate a mobile AR app in a CH space, in particular the Roman Ruins of Conimbriga. This evaluation, obtained through visitors from the mentioned archaeological site, supplied information related to users’ opinions regarding the advantages of using this technology in CH, as well as the most common issues related to the AR application performance.

The usefulness of this work concerns a better understanding for use of a mobile AR app in a CH context as a new way to enhance experiences and to provide information to users among CH scenarios and, in particular, among archaeological sites.

The virtual information displayed by the mobile AR app, DinofelisAR, was a virtual reconstruction of the ancient Forum, currently in ruins in Conimbriga. Held in the center of the Forum’s square, this study was achieved from a total of 90 visitors. Using a smartphone to run the DinofelisAR app, these participants were able to observe and explore a virtual reconstitution of the Forum superimposed on its ruins, in real-time, in 360 degrees. At the end of the experience, the volunteers filled-in a questionnaire, covering advantages of the usage of DinofelisAR, as well as related to their experience regarding the technological performance.

The results emphasize that AR technology brings added value to CH by benefiting visitors and institutions. Even though the performance issues of this technology did not hamper visitors’ experience, there are some improvements that could be made. Therefore, this study expresses the great potential of AR applications such as the DinofelisAR prototype, prompting relevant and opportune new lines of research among scientific community, for cultural legacy, and for society in general.

For future work, some technological approaches may be reconsidered to achieve better AR solutions for outdoor spaces, in order to rectify some issues and limitations presented. Looking ahead, deeply understanding on users’ intention to use AR technology is necessary to accomplish better approaches that would fit visitors’ expectations when visiting CH sites, contributing this way for the consolidation of AR solutions in CH sites.