Next Article in Journal
Donate Time or Money? The Determinants of Donation Intention in Online Crowdfunding
Previous Article in Journal
Evaluation of Empirical Reference Evapotranspiration Models Using Compromise Programming: A Case Study of Peninsular Malaysia
Previous Article in Special Issue
Transforming Communication Channels to the Co-Creation and Diffusion of Intangible Heritage in Smart Tourism Destination: Creation and Testing in Ceutí (Spain)
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sustainability
Volume 11
Issue 16
10.3390/su11164268
sustainability-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Augmented Reality Markerless Multi-Image Outdoor Tracking System for the Historical Buildings on Parliament Hill

by
Silvia Blanco-Pons
1,*,
Berta Carrión-Ruiz
1,
Michelle Duong
2,
Joshua Chartrand
2,
Stephen Fai
2 and
José Luis Lerma
1
1
Photogrammetry & Laser Scanning Research Group (GIFLE), Department of Cartographic Engineering, Geodesy and Photogrammetry, Universitat Politècnica de València, 46022 Valencia, Spain
2
Carleton Immersive Media Studio, Carleton University, Ottawa, ON K1S 5B6, Canada
*
Author to whom correspondence should be addressed.
Sustainability 2019, 11(16), 4268; https://doi.org/10.3390/su11164268
Submission received: 31 May 2019 / Revised: 22 July 2019 / Accepted: 30 July 2019 / Published: 7 August 2019
(This article belongs to the Special Issue Cultural Heritage and Smart Tourism)
Browse Figures
Versions Notes

Abstract

:
Augmented Reality (AR) applications have experienced extraordinary growth recently, evolving into a well-established method for the dissemination and communication of content related to cultural heritage—including education. AR applications have been used in museums and gallery exhibitions and virtual reconstructions of historic interiors. However, the circumstances of an outdoor environment can be problematic. This paper presents a methodology to develop immersive AR applications based on the recognition of outdoor buildings. To demonstrate this methodology, a case study focused on the Parliament Buildings National Historic Site in Ottawa, Canada has been conducted. The site is currently undergoing a multiyear rehabilitation program that will make access to parts of this national monument inaccessible to the public. AR experiences, including simulated photo merging of historic and present content, are proposed as one tool that can enrich the Parliament Hill visit during the rehabilitation. Outdoor AR experiences are limited by factors, such as variable lighting (and shadows) conditions, caused by changes in the environment (objects height and orientation, obstructions, occlusions), the weather, and the time of day. This paper proposes a workflow to solve some of these issues from a multi-image tracking approach.

1. Introduction

This paper presents a smart tourism application based on Augmented Reality (AR) for real-time visualization of the exterior of the Parliament Hill buildings, one of the most important cultural sites of Ottawa. AR technology is currently being applied in a number of disciplines and has demonstrated the capacity to deliver a palpable sense of immersion in the user. Typically, AR tracking is based on markers or known images but in some cases, such as outdoor tracking, these solutions cannot be used. Outdoor environments change constantly due to factors such as weather conditions, sunlight, or human activity. This often makes it difficult to obtain a unique image for tracking or using a marker that can last over time.
This study is focused on the development and use of a set of mobile AR solutions in outdoor scenarios at the Parliament Buildings National Historic Site of Canada (Figure 1a–c). The site is comprised of four buildings—Centre Block, West Block, East Block, the Library of Parliament—and the grounds. This site—known colloquially as “Parliament Hill”—is a major attraction for tourists who visit Ottawa and a popular gathering place for local residents.
As seen in Figure 1d, the four buildings on Parliament Hill flank a central green space. At the southern end sits the Centennial Flame monument, a well-known landmark and meeting point. This area is open and accessible to the public. Since the buildings sit on a promontory, the view beyond Centre Block is open sky. While dramatic as an architectural setting, dynamic changes in foreground figuration and background illumination can be problematic for AR tracking. This study explores solutions for outdoor tracking in this type of environment, evaluating the multi-image tracking approach from current tracking solutions, and more specifically, the image recognition approach of Vuforia Library.
This paper demonstrates the possibility of developing functional AR apps for the recognition of the building facades directly affected by factors such as occlusions and large variations in lighting using a multi-image tracking approach. By previously evaluating the changes in light and shadows that affect the buildings, it is possible to prepare appropriate images for the recognition of the buildings for any time of the day and in different seasons.
This paper is organized as follows. Section 2 presents existing works regarding tracking, AR and the dissemination of cultural heritage. Section 3 describes the proposed markerless tracking system and gives details about tracking challenges to evaluate the system. Section 4 discusses the results obtained in the evaluations performed. Section 5 draws conclusions and suggests future work.

2. Related Work

Many studies have been conducted in the cultural heritage field with AR technology. This section describes current state-of-the art solutions for outdoor tracking for cultural heritage, as well as the 3D data optimization required to develop an AR app.

2.1. AR Experiences in Cultural Heritage

Azuma [1] defines AR as a system that layers interactive virtual objects on real environments in real time, so that users perceive the virtual objects to be part of the real world. To achieve a proper immersive experience, the system must obtain a high level of accuracy for position and orientation of the user’s device so that the real and virtual objects are perfectly aligned.
The use of immersive tools to enhance access to cultural heritage is well established [2]. In recent years, AR systems have been integrated into museums, gallery exhibitions, and guided tours to improve visitor experience. Gimeno et al. [3] present a novel, mobile augmented guide for the Casa Batlló museum (Barcelona, Spain). Kolivand et al. [4] describe ReVitAge, an AR system that shows realistic reconstructed heritage buildings in real environments. Amakawa et al. [5] developed The New Philadelphia AR Tour, a mobile application that allows the visitor to walk through historical building reconstruction using AR. Kim et al. [6] present an AR tour application that allows visitors to interact with the tour exhibitions and improve communication. Barrile et al. [7] present an AR application that integrates information on the ground and underground to identify buried structures on-site.

2.2. Tracking Outdoor

Several tracking approaches have been studied in the last decade and can be divided into two groups: marker-based tracking and markerless tracking. The first is based on the recognition of fiducial markers [8,9] and the latter is based on the recognition of natural features and points that are visible in the real environment [10].
Tracking methods have been applied in many cases and recognition algorithms are continuously improving. In the case of outdoor AR applications there are some restrictions that hinder the tracking process such as occlusions, large variations in lighting, the impossibility of modifying the environment, as well as an ever-changing environment due to unpredictable weather conditions, pollution, or physical changes to the environment. These challenges have led many studies to focus on solving these problems.
One of the first studies of outdoor AR applications was Azuma et al. [11] who combined different sensors to obtain the position and orientation of a device. Cirulis et al. [12] developed a solution to merge a real city with virtual buildings in outdoor environments. In this case, the authors used GPS to obtain the user’s location, a gyroscope for head orientation, and a digital compass for sight direction detection. These solutions suffer from poor registration because the device sensors do not have the accuracy required for AR tracking. In particular, GPS accuracy decreases in urban environments due to signal reflections.
To improve tracking accuracy, other researchers have focused on hybrid solutions. Reitmayr et al. [13] present a robust hybrid tracking system for outdoor AR in urban environments. Their system combines edge-based tracking extended to a textured 3D model with inertial and magnetometer sensors. You et al. [14] combined vision-based tracking with inertial and compass sensors to compensate for the weaknesses of each approach.
Wither et al. [15] propose a new approach called “indirect AR”, which achieves perfect alignment between real and virtual worlds in outdoor scenarios. They replace the live camera view used in AR with a previously captured panoramic image where the virtual content is placed. The main problem is that some elements of the scene, such as weather, lighting, and vehicles, may not be represented correctly and the user may not feel fully immersed. The results of this study indicate that, in most conditions, users preferred indirect AR over traditional AR. Okura et al. [16], looking to improve the sense of realism, propose a system that takes into account real-world illumination, selecting the proper image from a set of pre-captured images under different weather conditions and times of day. Gimeno et al. [3] applied this method in a museum where the small and crowded spaces made it impossible to use the feature recognition approach. While effective, this approach requires a live camera view that allows the user to take a photo with the augmented object.
The vision-based approach is another markerless solution that relies on computer vision and provides very accurate tracking, although it can be unstable as it depends on natural-feature detection. This method is explained in several papers [17,18,19,20] and is based on keypoint extraction and feature descriptors to calculate the pose estimation. This approach has been applied for the augmentation of the urban environment [21] or in object detection in uncontrolled outdoor environments [22].

2.3. 3D Data Optimization

The capabilities of the majority of mobile devices force developers to optimize media content as much as possible. This results in the manipulation of content, whether they are images, videos, animations, or 3D models. The production of optimized, automated 3D content is an important challenge that many researchers have studied in-depth. These challenges include mesh optimization or remeshing, control of the Level-Of-Detail (LOD), and compatibility between format files.
In regards to mesh optimization, the goal is to reduce the size of the mesh while maintaining the required LOD in each use case. When simplifying and improving the quality of the meshes, the geometry and the topology of the mesh have to be taken into account and many algorithms have been developed for this purpose [23,24,25,26,27,28]. However, it is important to note that the success of the method will depend on the shape of the element one is dealing with. Therefore, the mesh optimization process must be appropriate for each specific case.
Controlling the LOD—the visualization of unnecessary details or invisible faces in the scene of the application—can help save memory. Thus, anything that is not visible to the user is not shown. For attaining these outcomes, Guidi et al. [29] control the LOD by employing multi-resolution modeling sources. Some authors try to control the LOD by simplifying the models in real time. For an in-depth review of real-time rendering, see Akenine-Möller et al. [30].
Finally, as Remondino and El- Hakim [31] have pointed out, over the years, many different 3D graphics file formats have been developed. While format standardization would make the exchange of 3D data easier, the most commonly used formats are available in the majority of software.

3. Materials and Methods

3.1. Mobile Device

The mobile device employed for the multi-image target testing is detailed in Table 1.

3.2. Outdoor Tracking

Tracking is an important issue in a real-time AR context and requires a high level of precision and low latency. Camera precision is required so that the augmented content is properly aligned with the real-time view in the camera. Low latency assures that the precision is maintained even if there is a slight movement of the camera. Proper tracking provides a seamless integration of “the real” and “the virtual”. This paper proposes a feature-based approach using natural feature points identified from images taken in real time through the camera of a mobile device. Virtual content is displayed only when the device detects these feature points. This type of tracking poses challenges in outdoor environments.
When it comes to developing an outdoor AR experience, numerous problems can arise that will cause poor tracking, virtual content flicker, or even tracking loss. Accordingly, it is desirable to test this type of tracking in AR apps to find out what the implicating factors are and to determine the best possible solutions.
A common way to accomplish successful AR is by detecting planar (fiducial) markers placed on the object and/or in the landscape. This is problematic in the context of historic buildings where markers are considered intrusive in most cases. Consequently, in cases involving historic buildings, it is more suitable to rely on permanent features present in the real world and use image based tracking. This approach is less intrusive and the user can perceive the augmented elements and interact with them normally with minimum effort. However, some additional requirements must be taken into account in order to use an outdoor image tracking solution in an AR application.
The accurate position of the content depends on the detection and tracking of features that are found in the image target. The camera pose is calculated from natural features that are extracted from the image target and then compared at run time with features in the live camera image. Markerless methods exploit the natural features existing in the real scene such as the corners and edges. Therefore, to create a successful AR experience that is accurately detected, images that are rich in detail, with good contrast, and no repetitive patterns should be used. Fortunately, the Parliament Hill buildings are very detailed and provide image targets that will produce accurate tracking.
The user must be placed approximately at the same point as where the image target was taken in order to calculate the camera pose, and consequently to activate the augmented experiences. On Parliament Hill, the image targets have been taken from accessible vantage points for visitors that coincide with the pose required for image target detection (Figure 1d), using one standard location for each experience where possible.
Furthermore, there is no control over the potential changes in the environment, such as occlusions derived from added elements of the rehabilitation or on-site events. As such, it is necessary to plan in detail the capture of target images and to avoid elements that can cause tracking malfunctions.
In addition, changes in lighting conditions can cause drastic changes in the appearance of certain objects. For example, the orientation of the facades of the buildings and the changes in the ambient light of Parliament Hill cause changes in the shadows that are cast on the buildings. Therefore, the appearance of the building facades varies depending on the time of day, the time of year, or weather conditions. All of these variations can cause a launch failure of the augmented content. In order to address this issue a shadow area study was carried out (cf. Section 3.3).
Considering the requirements for successful AR tracking discussed above and taking a close look at the shadow studies, a vision-based approach was established and developed using the Vuforia AR library. This library is easier to use than others, such as ARCore, and it includes the multi-image approach and a free developer license. In addition, since shadows drastically affect image recognition, a multitarget approach was required. Various images of the target were implemented into one scene in order to increase the detection accuracy.

3.3. Multitarget Outdoor Testing on Parliament Hill

The study was conducted during the summer months. It was necessary to predict the possible changes in the position of the sun since the changes in the lighting and shadows on the facades can cause a malfunction of the tracking system. Therefore, a light and shadow study on West Block, Centre Block and East Block 3D models has been carried out to show the position of the sun throughout the year (Figure 2, Figure 3 and Figure 4, respectively):
Taking into account the orientation of the site and the sun light mapping simulation, a set of images for the recognition of each building was taken (image targets). The shadows especially affect the West Block and Centre Block facades, as seen in Figure 5 and Figure 6; thus, in these cases, three and four image targets have been used, respectively. As seen in the shadow study of the East Block, the sun light does not affect this façade, thus, one image target has been used (Figure 7).
Finally, to verify the correct recognition of the buildings, several tests were carried out at different times of the day and with different environmental conditions. To that end, a script to determine which image target was recognized at any time was developed. This information is summarized in Table 2.
  • West Block: From 18:00 on, the sun is behind the building. Therefore, there are no shadows but from 7:00 to 18:00 the orientation of the sun causes significant change to the image of the building (Figure 2). All tests performed on a sunny day detected the targets correctly. On cloudy days there are no shadows on this building; for this reason, the sunny 18:00 target works well in all the test cases on cloudy days and during sunset (Table 2).
  • Centre Block: As in the case of West Block, during the day the sun changes the building images so four image targets were used (Table 2). From 18:00 on, the sun is behind the building, therefore, there are no shadows and the building image is the same as on a cloudy day (Figure 3).
  • East Block: The orientation of the facade of this building is not affected by the sun at any time of the day (Figure 4). Therefore, only one target was required and in all cases tested, recognition and tracking worked well (Table 2).

3.4. Realism

It is essential to maintain a sense of realism in an AR app in order to provide a seamless experience. This is accomplished by ensuring the user feels that the digital objects belong to the real world, thus 3D content should be as visually consistent with the real world as possible. However, how realistic an object appears must be balanced with file size. For example, in the case of photogrammetric 3D modeling, the fidelity of the final 3D model can be very high, but the size of the generated file may be too large for mobile apps [32,33,34]. Current practice makes use of remeshing techniques and the addition of normal maps to reduce the size of the mesh while maintaining the realism of the final 3D model. By incorporating realistic 3D models and additional interactivity to the AR experience, the user can feel more immersed in the app.
1. Remeshing architectural features.
It is possible to create extremely high quality 3D models using photogrammetry. However, the generated 3D models contain a very large number of polygonal faces, resulting in meshes that are too heavy and complex for loading and displaying in mobile devices. Therefore, the recreation and optimization of the same polygonal surface with more optimal geometry by retopologizing the mesh is necessary for AR app development. Instant Meshes, an open source, quad-based autoretopology software has been used herein to simplify complex meshes faster and easier than other workflow methodologies [32].
2. Texturing grotesques.
The goal in retexturing meshes is to make a low-resolution mesh as visually similar as possible to a high-resolution mesh using a set of processes that transfer details from one model to another (the process is known as baking). Baking a mesh will retain detailed information from a high-resolution mesh that is otherwise lost in a low-resolution version [35].
Normal maps can project the geometry of a high-resolution mesh onto a low-resolution mesh. As such, the combination of two texture layers, one being a colored texture and the second being a normal map, can be utilized to add detail at different angles. In this case the texture of the photogrammetric 3D model and the normal map were used to produce a mesh that resembles a highly photorealistic photogrammetric 3D model, even though it is a low-resolution mesh.
3. Interactivity.
Typically, AR applications rely on passive experiences in which the user points the camera of a mobile device to the scene and virtual objects are displayed. However, a more active experience can be developed through greater interaction with the app and device. As such, for this study, different actions have been developed to display and modify the virtual content using Unity, a powerful cross-platform game engine with the ability to manipulate 3D content using C# scripting. This software, compared to others, is the easiest to use [36] and Vuforia Library provides a plugin for it.
The intent of the AR experiences at the Centre Block is to show visitors the history of the building. The first Centre Block that was built in the late 1800s was destroyed by a fire in 1916. While the structure that was built to replace it has a similar building footprint and symmetrical facade, the second Centre Block is taller and has a more predominant central element—the Peace Tower. The Peace Tower is the terminus of a long central axis of the Parliamentary grounds.
To show the architectural differences between the original Centre Block (Figure 8b) and the Centre Block standing on Parliament Hill today (Figure 8a), a set of archival images was used to create an animation of the construction process over the last century. When the application recognizes the current Centre Block building, it presents the first archival image, after which the user can view the rest of the images by swiping the touch screen (Figure 8). Additionally, pressing a button can activate or deactivate the animation.
Another important feature of the application is the possibility for the user to take a photo with the historical content in the background, similar to a modern-day ‘selfie’. To accomplish this, a button has been added that takes a screenshot and stores it in the device’s gallery using the Unity Native Gallery plugin. The virtual content that arose is always displayed in front of the real world, that is, virtual images appear in front of everything, covering the people intended to be part of the photo being taken. To solve the occlusion problem, the virtual images were placed above the visitors, which was possible due to the position of the building as seen in Figure 8. In addition, the background frame was changed to transparent, thus avoiding covering part of the subject’s body. Once the AR experience is launched at the Centre Block (Figure 1a), it allows the virtual content to appear above the subject in the photo.
On the West Block, there are two grotesques placed on the East facade of the building situated on either side of the entrance to the East Wing. As an attempt to better showcase appreciation for these grotesques, an AR experience was developed that presents two interactive hotspots (Figure 9) which, upon being tapped, open up a new display with an animation of the 3D grotesque selected. In addition, the user can zoom in and zoom out to see the detail of each 3D model.

4. Discussion

The image recognition presented herein works well due to the flat nature of the buildings’ facades. However, the AR app must be launched from a specific location in order to maintain the same geometry between images captured by the user and the image used as a markerless target. Since image recognition is based on a set of geometric features from the stored image, the image must be the same. This remains problematic. The orientation of the sun creates very harsh shadows that change depending on the time of day, producing different images of the building. For this reason, tracking in the afternoon does not work with an image target taken in the morning. However, by adding different image targets of different times of the day, this problem was solved. From 18:00 on, the sun is behind the Hill and there are no shadows on the façades, just like on a cloudy day. If the facade is not affected by the sun, as in the case of East Block, the application works with one image target regardless of the time of the day.
This application was developed during the summer. For other seasons, it would be necessary to add additional images.
Another thing to consider is the realism of the virtual content. According to Bruno et al. and Comes et al. [33,34], visual realism depends on two components: first, the capacity of the virtual object to appear real in contrast to the geometry and texture of the model and, second, the fidelity of the lighting. In the case presented herein, the images superimposed on the Centre Block have been previously processed, eliminating the background and have been placed and scaled to obtain a good alignment with the real building so that the user perceives the image to be as real as possible. The problem with the Vuforia library is that it does not produce shadows when the virtual content is added. This affected the quality of the grotesque experience in particular. The issue could be addressed by using other libraries that compensate for shadows—such as ARCore.
The main goal of an AR application is to display virtual and real objects together accurately with minimal registration error so that the user feels that the virtual objects are part of the real scene. The problem that arises is that virtual objects are always displayed in front of the real objects causing a lack of realism. This problem arises when the user wants to take a selfie with the augmented content. When the user partially blocks the view of the virtual content, it does not work properly as it should be hidden away to make it appear behind the user. To avoid the occlusion issue in this case study, a transparent background frame has been added to the image (virtual content) so that it does not cover users who want to take selfies. In addition, due to the building position, the augmented image is typically above the user, thus avoiding the occlusion problem. This method works flawlessly for users of average height, so a more appropriate solution would be to use a real-time method to calculate the occlusion. Currently, occlusion issues are a major challenge in AR and several studies are focused on solving this problem. Behzadan et al. [37] introduce a depth-based occlusion handling method to detect and correct occlusion cases between virtual and real objects in real time and in an outdoor, unprepared environment. Yuan et al. [38] propose an effective occlusion handling method based on 3D reconstruction. Other research studies follow a contour-based approach to resolve the occlusion problem in AR [39,40]. The problem with these solutions is that they require significant processing power. This may be less of a problem in the future.
One of the advantages of an AR app is the ability to display large amounts of information related to an object and to access that information interactively. This can be used to enhance the tourist experience [41] or to augment learning experiences in context of structured education [42].

5. Conclusions

In this paper, outdoor AR tracking has been studied using a markerless multi-image approach in a real environment. The markerless multi-image approach is characterized by using of a set of images taken at different times of the day and taken from the same position. The results demonstrate that this approach is effective in an outdoor environment with dramatic changes in lighting, thus this methodology could be applied in most cases if a thorough study of the lighting is conducted beforehand.
The advantages of this markerless multi-image approach over others, such as model-based tracking, are faster processing time and the availability of free libraries (i.e., as provided by Vuforia and ARCore) that include image tracking. Multi-image tracking is a new way to achieve a functional AR application in outdoor environments, being fast and easy to apply. Despite the increased number of AR applications that are being developed and applied to smart tourism, especially in the field of cultural heritage, these applications are generally focused on museum exhibitions due to the difficulty to achieve adequate results in outdoor environments, as it has been reported in this paper.
In future work additional imagery will be added to enrich the markerless image dataset (i.e., pictures with snow). This multi-image approach will be compared with model-based tracking to identify the best AR solution for outdoor environments.

Author Contributions

S.B.-P. conceived of the presented idea; S.B.-P. and B.C.-R. implemented the AR apps and carried out the experiment and outdoor tests, in addition to writing the manuscript; M.D. and J.C. designed the shadow study and contributed to the final version of the manuscript; S.F. and J.L.L. supervised the project and review the manuscript.

Funding

This work has been developed under the framework of the New Paradigms/New Tools for Heritage Conservation in Canada, a project funded through the Social Sciences and Humanities Research Council of Canada (SSHRC).

Acknowledgments

This work has been developed under the framework of the New Paradigms/New Tools for Heritage Conservation in Canada, a project funded through the Social Sciences and Humanities Research Council of Canada (SSHRC).

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Azuma, R. A survey of augmented reality. Presence Teleoper. Virtual Environ. 1997, 6, 355–385. [Google Scholar] [CrossRef]
  2. Bekele, M.K.; Pierdicca, R.; Frontoni, E.; Malinverni, E.S.; Gain, J. A Survey of Augmented, Virtual, and Mixed Reality for Cultural Heritage. J. Comput. Cult. Herit. 2018, 11, 1–36. [Google Scholar] [CrossRef]
  3. Gimeno, J.J.; Portalés, C.; Coma, I.; Fernández, M.; Martínez, B. Combining traditional and indirect augmented reality for indoor crowded environments. A case study on the Casa Batlló museum. Comput. Graph 2017, 69, 92–103. [Google Scholar] [CrossRef]
  4. Kolivand, H.; Rhalibi, A.; Sunar, M.S.; Saba, T. ReVitAge: Realistic virtual heritage taking shadows and sky illumination into account. J. Cult. Herit. 2018, 1–10. [Google Scholar] [CrossRef]
  5. Amakawa, J.; Westin, J. New Philadelphia: Using augmented reality to interpret slavery and reconstruction era historical sites. Int. J. Herit. Stud. 2018, 24, 315–331. [Google Scholar] [CrossRef]
  6. Kim, J.B.; Park, C. Development of mobile AR tour application for the national palace museum of Korea. Lect. Notes Comput. Sci. 2011, 6773, 55–60. [Google Scholar] [CrossRef]
  7. Barrile, V.; Fotia, A.; Bilotta, G.; Carlo, D. Integration of geomatics methodologies and creation of a cultural heritage app using augmented reality. Virtual Archaeol. Rev. 2019, 10, 40–51. [Google Scholar] [CrossRef]
  8. Pentenrieder, K.; Meier, P.; Klinker, G. Analysis of Tracking Accuracy for Single-Camera Square-Marker-Based Tracking. In Third Workshop on Virtual and Augmented Reality of the GI-Fachgruppe VR/AR, Koblenz, Germany, 2006. Available online: http://campar.in.tum.de/Chair/PublicationDetail?pub=pentenrieder2006gi (accessed on 1 August 2019).
  9. Kato, H.; Billinghurst, M. Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality, San Francisco, CA, USA, 20–21 October 1999. [Google Scholar]
  10. Herling, J.; Broll, W. Markerless Tracking for Augmented Reality. In Handbook of Augmented Reality; Furht, B., Ed.; Springer: New York, NY, USA, 2011. [Google Scholar]
  11. Azuma, R.; Hoff, B.; Neely, H.; Sarfaty, R. A motion-stabilized outdoor augmented reality system. In Proceedings of the IEEE Virtual Reality (Cat. No. 99CB36316), Houston, TX, USA, 13–17 March 1999. [Google Scholar]
  12. Cirulis, A.; Brigmanis, K.B. 3D outdoor augmented reality for architecture and urban planning. Procedia Comput. Sci. 2013, 25, 71–79. [Google Scholar] [CrossRef]
  13. Reitmayr, G.; Drummond, T. Going out: Robust model-based tracking for outdoor augmented reality. In Proceedings of the 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality, Santa Barbard, CA, USA, 22–25 October 2006. [Google Scholar]
  14. You, S.; Neumann, U.; Azuma, R. Orientation tracking for outdoor augmented reality registration. IEEE Comput. Graph. Appl. 1999, 19, 36–42. [Google Scholar] [CrossRef]
  15. Wither, J.; Tsai, Y.T.; Azuma, R. Indirect augmented reality. Comput. Graph. 2011, 35, 810–822. [Google Scholar] [CrossRef]
  16. Okura, F.; Akaguma, T.; Sato, T.; Yokoya, N. Indirect augmented reality considering real-world illumination change. In Proceedings of the 2014 IEEE International Symposium on Mixed and Augmented Reality, Munich, Germany, 10–12 September 2014. [Google Scholar]
  17. Sörös, G.; Seichter, H.; Rautek, P.; Gröller, E. Augmented visualization with natural feature tracking. In Proceedings of the 10th International Conference on Mobile and Ubiquitous Multimedia, Beijing, China, 7–9 December 2011. [Google Scholar]
  18. Uchiyama, H.; Marchand, E. Object Detection and Pose Tracking for Augmented Reality: Recent Approaches. In Proceedings of the 18th Korea-Japan Joint Workshop on Frontiers of Computer Vision, Kawasaki, Japan; 2012; pp. 1–8. Available online: https://www.researchgate.net/publication/264846831_Object_Detection_and_Pose_Tracking_for_Augmented_Reality_Recent_Approaches (accessed on 1 August 2019).
  19. Radkowski, R.; Oliver, J. Natural feature tracking augmented reality for on-site assembly assistance systems. In Lecture Notes in Computer Science; Shumaker, R., Ed.; Springer: Berlin/Heidelberg, Germany, 2013. [Google Scholar]
  20. Moreels, P.; Perona, P. Evaluation of Feature Detectors and Descriptors based on 3D Objects. Int. J. Comput. Vis. 2007, 73, 263–284. [Google Scholar] [CrossRef]
  21. Carozza, L.; Tingdahl, D.; Bosché, F.; Gool, L. Markerless vision-based augmented reality for urban planning. Comput. Civ. Infrastruct. Eng. 2014, 29, 2–17. [Google Scholar] [CrossRef]
  22. Rao, J.; Qiao, Y.; Ren, F.; Wang, J.; Du, Q. A Mobile Outdoor Augmented Reality Method Combining Deep Learning Object Detection and Spatial Relationships for Geovisualization. Sensors 2017, 17, 1951. [Google Scholar] [CrossRef] [PubMed]
  23. Hoppe, H.; Derose, T.; Mcdonald, J.; Stuetzle, W.; Duchamp, T. Mesh Optimization. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques, Anaheim, CA, USA, 2–6 August 1993. [Google Scholar] [CrossRef]
  24. Rossignac, J.; Borrel, P. Multi-resolution 3D approximations for rendering complex scenes. Model. Comput. Graph. 1993, 455–465. [Google Scholar] [CrossRef]
  25. Gross, M.H.; Member, A.; Staadt, O.G.; Gatti, R. Efficient Triangular Surface Appr Using Wavelets and Quadtree Dat. IEEE Trans. Vis. Comput. Graph. 1996, 2, 130–143. [Google Scholar] [CrossRef]
  26. Botsch, M.; Pauly, M.; Rossl, C.; Bischoff, S.; Kobbelt, L. Geometric modeling based on triangle meshes. In Proceedings of the ACM SIGGRAPH 2006 Courses, Boston, MA, USA, 30 July–3 August 2006. [Google Scholar] [CrossRef]
  27. Pietroni, N.; Tarini, M.; Cignoni, P. Almost isometric mesh parameterization through abstract domains. IEEE Trans. Vis. Comput. Graph. 2009, 16, 621–635. [Google Scholar] [CrossRef]
  28. Khan, D.; Yan, D.M.; Ding, F.; Zhuang, Y.; Zhang, X. Surface remeshing with robust user-guided segmentation. Comput. Vis. Media 2018, 4, 113–122. [Google Scholar] [CrossRef] [Green Version]
  29. Guidi, G.; Remondino, F.; Russo, M.; Menna, F.; Rizzi, A.; Ercoli, S. A Multi-Resolution Methodology for the 3D Modeling of Large and Complex Archeological Areas. Int. J. Archit. Comput. 2009, 7, 39–55. [Google Scholar] [CrossRef]
  30. Akenine-Möller, T.; Haines, E.; Hoffman, N. Real-Time Rendering, 3rd ed.; A K Peters: Natick, MA, USA; CRC Press: Boca Raton, FL, USA, 2008. [Google Scholar]
  31. Remondino, F.; El-Hakim, S. Image-Based 3D Modelling: A Review. Photogramm. Rec. 2006, 21, 269–291. [Google Scholar] [CrossRef]
  32. Zhu, L.; Hyyppa, J.; Chen, R.; Liu, Z. An approach of 3D model simplification for mobile phone based navigation application. In Proceedings of the 2010 Ubiquitous Positioning Indoor Navigation and Location Based Service, Kirkkonummi, Finland, 14–15 October 2010; pp. 2–5. [Google Scholar]
  33. Bruno, F.; Bruno, S.; Sensi, G.; Luchi, M.L.; Mancuso, S.; Muzzupappa, M. From 3D reconstruction to virtual reality: A complete methodology for digital archaeological exhibition. J. Cult. Herit. 2010, 11, 42–49. [Google Scholar] [CrossRef]
  34. Comes, R.; Neamțu, C.; Buna, Z.; Badiu, I.; Pupeză, P. Methodology to Create 3D Models for Augmented Reality Applications Using Scanned Point Clouds, Mediterr. Archaeol. Archaeom. 2014, 14, 35–44. [Google Scholar]
  35. Unity, The Photogrammetry Workflow. 2018. Available online: https://unity.com/solutions/photogrammetry. (accessed on 1 September 2018).
  36. Blanco-Pons, S.; Carrión-Ruiz, B.; Lerma, J.L. Review of Augmented Reality and Virtual Reality Techniques. In Proceedings of the 8th International Congress on Archaeology, Computer Graphics, Cultural Heritage and Innovation, Valencia, Spain, 5–7 September 2016. [Google Scholar] [CrossRef]
  37. Behzadan, A.H.; Kamat, V.R. Scalable algorithm for resolving incorrect occlusion in dynamic augmented reality engineering environments. Comput. Civ. Infrastruct. Eng. 2010, 25, 3–19. [Google Scholar] [CrossRef]
  38. Tian, Y.; Long, Y.; Xia, D.; Yao, H.; Zhang, J. Handling occlusions in augmented reality based on 3D reconstruction method. Neurocomputing 2015, 156, 96–104. [Google Scholar] [CrossRef]
  39. Berger, M.O. Resolving occlusion in augmented reality: A contour based approach without 3D reconstruction. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Juan, FL, USA, 17–19 June 1997. [Google Scholar]
  40. Tian, Y.; Guan, T.; Wang, C. Real-time occlusion handling in augmented reality based on an object tracking approach. Sensors 2010, 10, 2885–2900. [Google Scholar] [CrossRef]
  41. Shang, L.W.; Zakaria, M.H.; Ahmad, I. Mobile phone augmented reality postcard. J. Telecommun. Electron. Comput. Eng. 2016, 8, 135–139. [Google Scholar]
  42. Sunil, S.; Nair, S.S.K. An Educational Augmented Reality App to Facilitate Learning Experience. Int. Conf. Comput. Appl. ICCA 2017, 2017, 279–282. [Google Scholar] [CrossRef]
Sustainability 11 04268 g001 550
Figure 1. Parliament Hill (Ottawa): (a) History Centre Block AR experience located at the Centre Block, (b) grotesque AR experience located at the West Block, (c) OWL AR experience located at the East Block, and (d) geographic location of the buildings and AR experiences location.
Figure 1. Parliament Hill (Ottawa): (a) History Centre Block AR experience located at the Centre Block, (b) grotesque AR experience located at the West Block, (c) OWL AR experience located at the East Block, and (d) geographic location of the buildings and AR experiences location.
Sustainability 11 04268 g001
Sustainability 11 04268 g002 550
Figure 2. West Block Shadow Study at different times of day.
Figure 2. West Block Shadow Study at different times of day.
Sustainability 11 04268 g002
Sustainability 11 04268 g003 550
Figure 3. Centre Block Shadow Study at different times of day.
Figure 3. Centre Block Shadow Study at different times of day.
Sustainability 11 04268 g003
Sustainability 11 04268 g004 550
Figure 4. East Block Shadow Study at different times of day.
Figure 4. East Block Shadow Study at different times of day.
Sustainability 11 04268 g004
Sustainability 11 04268 g005 550
Figure 5. Image targets of West Block taken approximately 14 m away: (a) Sunny 8 AM, (b) Sunny 1 PM, and (c) Sunny 6 PM.
Figure 5. Image targets of West Block taken approximately 14 m away: (a) Sunny 8 AM, (b) Sunny 1 PM, and (c) Sunny 6 PM.
Sustainability 11 04268 g005
Sustainability 11 04268 g006 550
Figure 6. Image targets of Centre Block taken approximately 100 m away: (a) Sunny 8 AM, (b) Sunny 1 PM, (c) Cloudy 6 PM, and (d) Cloudy 9 PM.
Figure 6. Image targets of Centre Block taken approximately 100 m away: (a) Sunny 8 AM, (b) Sunny 1 PM, (c) Cloudy 6 PM, and (d) Cloudy 9 PM.
Sustainability 11 04268 g006
Sustainability 11 04268 g007 550
Figure 7. Image target of East Block taken approximately 10 m away.
Figure 7. Image target of East Block taken approximately 10 m away.
Sustainability 11 04268 g007
Sustainability 11 04268 g008 550
Figure 8. Centre Block: (a) Tourist photography (2018) and (b) simulated photography merging historic and present (2018) content as a deliverable of the History Centre Block AR app. This app was developed with the Vuforia AR Library.
Figure 8. Centre Block: (a) Tourist photography (2018) and (b) simulated photography merging historic and present (2018) content as a deliverable of the History Centre Block AR app. This app was developed with the Vuforia AR Library.
Sustainability 11 04268 g008
Sustainability 11 04268 g009 550
Figure 9. Screenshot taken with the Grotesque AR experience in which the hotspot appears.
Figure 9. Screenshot taken with the Grotesque AR experience in which the hotspot appears.
Sustainability 11 04268 g009
Table
Table 1. Mobile device specifications.
Table 1. Mobile device specifications.
SmartphoneOperating System (OS)Application Programming Interface (API)ProcessorScreenCameraMemory RAMStorage
Samsung S8
Sustainability 11 04268 i001		Android8.0 OreoExynos 9 Octa 8895 (2.3 GHz Quad + 1.7 GHz Quad CPU), GPU ARM Mali-G71 MP20 (546 MHz)5.8” 1.440 × 2.960 pixels 18.5:9 aspect ratio12MP AF, ƒ/1.7 Pixel size: 1.4µm Sensor size: 1/2.55” FOV: 77, Auto HDR, OIS (Optical Image Stabilization)
Sensor size: 1/3.6”
FOV: 80		4 GB RAM64 GB
Table
Table 2. Summary of the tests carried out at different times of the day on sunny and cloudy days at Parliament Hill buildings.
Table 2. Summary of the tests carried out at different times of the day on sunny and cloudy days at Parliament Hill buildings.
SUNNYCLOUDYSUNSET
WEST BLOCK
TARGET SUNNY 8 AM31-07-18 8:34
TARGET SUNNY 1 PM 16-08-18 13:20
TARGET SUNNY 6 PM 16-08-18 18:26 02-08-18 8:45 03-08-18 12:3030-07-18 18:15 09-08-18 20:30
CENTRE BLOCK
TARGET SUNNY 8 AM31-07-18 08:23
TARGET SUNNY 1 PM 16-08-18 13:20
TARGET CLOUDY 6 PM 16-08-18 18:18 02-08-18 08:05 03-08-18 12:3030-07-18 18:15 09-08-18 20:20
TARGET CLOUDY 9 PM 16-08-18 18:18 03-08-18 12:30 09-08-18 20:47
EAST BLOCK
TARGET CLOUDY 3PM31-07-18 08:20 16-08-18 13:35 16-08-18 18:3402-08-18 08:25 03-08-18 12:4930-07-18 18:20 09-08-18 20:39

Share and Cite

MDPI and ACS Style

Blanco-Pons, S.; Carrión-Ruiz, B.; Duong, M.; Chartrand, J.; Fai, S.; Lerma, J.L. Augmented Reality Markerless Multi-Image Outdoor Tracking System for the Historical Buildings on Parliament Hill. Sustainability 2019, 11, 4268. https://doi.org/10.3390/su11164268

AMA Style

Blanco-Pons S, Carrión-Ruiz B, Duong M, Chartrand J, Fai S, Lerma JL. Augmented Reality Markerless Multi-Image Outdoor Tracking System for the Historical Buildings on Parliament Hill. Sustainability. 2019; 11(16):4268. https://doi.org/10.3390/su11164268

Chicago/Turabian Style

Blanco-Pons, Silvia, Berta Carrión-Ruiz, Michelle Duong, Joshua Chartrand, Stephen Fai, and José Luis Lerma. 2019. "Augmented Reality Markerless Multi-Image Outdoor Tracking System for the Historical Buildings on Parliament Hill" Sustainability 11, no. 16: 4268. https://doi.org/10.3390/su11164268

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop