Augmented Reality- and Geographic Information System-Based Inspection of Brick Details in Heritage Warehouses

Abstract

Brick warehouses represent interdisciplinary heritage sites developed by social, cultural, and economic impacts. This study aimed to connect warehouse details and GIS maps in augmented reality (AR) based on the former Camphor Refinery Workshop Warehouse. AR was applied as an innovation interface to communicate the differences between construction details, providing a feasible on-site solution for articulating historical brick engineering technology. A complex warehouse cluster was georeferenced by the AR models of brick details. The map was assisted by a smartphone-based comparison of the details of adjacent warehouses. Sixty AR models of warehouse details exemplified the active and sustainable preservation of the historical artifacts. The side-by-side allocation of warehouse details in AR facilitated cross-comparisons of construction differences. We found that a second reconstructed result integrated AR and reality in a novel manner based on the use of a smartphone AR. GIS and AR facilitated a management effort using webpages and cloud access from a remote site. The vocabulary of building details can be enriched and better presented in AR.

1. Introduction

Augmented reality (AR) has been applied in numerous fields to facilitate high-fidelity interaction with reality. The realistic description of scenes enhances the comprehension of what is being presented within existing spatial structures (urban fabric). Should AR directly connect to mapping, specifically GIS mapping, to facilitate mutual benefits in 3D inspection? An interconnected AR- and GIS-based system should create advantages in multidisciplinary exploration, for example, the preservation of heritage warehouses.

One of the best approaches to exploring the historical context of a city is to inspect its industry-related constructions. Industrial buildings such as warehouses contribute to the spatiotemporal background and geographic distribution, under the impact of technology and culture. Most industrial buildings assembled using bricks were designed to process raw materials or store goods. Such constructions represent an entry point for understanding Taiwan’s early history and the degeneration of regional industry.

Elucidating the relationship between warehouses and urban fabric can deepen our knowledge of tangible heritage from different stages of development. Small buildings in relatively large urban areas indicate a slow pace of evolvement within the surrounding fabric. Such interactions have accelerated fabric degeneration or reactivation since 1880. Old warehouses have become part of the cultural landscape and heritage.

1.1. Research Goal and Scope

This study aimed to connect GIS maps and warehouse details in AR based on the former Camphor Refinery Workshop Warehouse. AR was applied as an interface to communicate the differences between construction details, providing a feasible on-site solution to articulate historical brick engineering technology. A complex warehouse cluster should be georeferenced by the AR models of brick details. The map was assisted by a smartphone-based comparison of details across adjacent warehouses on site.

All the brick warehouses are aged, have been reassembled by construction, or have been reactivated at different levels. The former Camphor Refinery Workshop Warehouse is an industry building that has close connections with railway transportation (Figure 1). This study explores the spatiotemporal interrelationships between the warehouse and the historical development of the urban fabric in Taipei, Taiwan. The scope of this study covers the scale of regional urban fabric to warehouse brick details. The study range started from the Quin Dynasty to the present day in Taipei. The fabric covers the traditional municipal area around the warehouse. Compared to the current fabric, this region gradually evolved, with the addition of new streets, railroads, and buildings, from being under development in 1930 (Figure 1c) to fully developed in 1983 (Figure 1d). The small circular gray dots in the figure below represent where the brick details were originally located in the old fabrics.

This study mainly utilized AR, and the novelty of this study lies in AR’s connection to GIS mapping. In terms of GIS mapping, a limited 3D map view can be published on, and accessed via, a webpage. Some of the details look similar and need to be placed side by side to conduct one-to-one comparisons. GIS annotates a more precise location of the details, becoming confused when the number reaches 60.

1.2. Related Studies

Virtual AR models that represent subjects of interest have to be sufficiently detailed to attract users or to deliver specific instructions. Photogrammetric modeling has gained attention in applications requiring the creation of photorealistic 3D models. Structure-from-Motion (SfM) photogrammetry was used to capture complex topography and assess volume via an inexpensive, effective, and flexible approach [2] with minimal expensive equipment or specialist expertise [3]. Applications can be found in wood [4,5], archeology [6], cultural heritage [7], architecture [8], construction [9], and construction progress tracking [10]. For large areas and those with an irregular shape, airborne photogrammetry and modeling has been applied to adapt architectural configurations to the slopes of sites [11] or to assess stone excavation [12]. In another study, a geospatial survey associated with UAV technology [13] was used to assess the percentage of cover and mean height of pine saplings [5]. Photogrammetry can be applied for small regular objects in indoor scenes [14]. In terms of using consumer-grade digital cameras for the low-cost acquisition of 3D data [15], the performance capabilities of today’s smartphone cameras enable this, as smartphones are now considered promising devices that can be used to capture imagery.

AR applications have been proven to be feasible for varied cases [16,17,18,19,20] of different scales [21,22], improving awareness of both the location and its context [23]. It should be noted that the documentation is related to information systems [24] and is part of a communication model [25] for exploration and dissemination [26]. Engagement and learning is of concern [27].

A synchronized first-person view offers the same perspective and enables video conferencing or online meetings across devices or platforms. In terms of remote collaboration, different AR system approaches have been developed to improve communication efficiency by applying gaze visualization platforms for physical tasks [28], 3D gesture and CAD models of mixed reality (MR) for training tasks [29], remote collaboration [30], image- or live-video-based AR collaborations [31], remote diagnosis [32], 360° video cameras in augmented telecollaboration [33], web-based extended reality (XR) [34], 2D/3D telecollaboration [35], and indoor construction monitoring [36].

Urbanization, land use alteration, and loss of functionality are factors that jeopardize heritage. The definitions and categories of cultural landscapes have been discussed worldwide [37]. Warehouses are recognized as industrial landscapes [38] that affect people, places, and the environment over time [39]. Traditional industrial building types are evidence of vanishing traditions [40]. Warehouses are hardly mentioned as part of the architectural culture unless they take the form of factories [41]. The question of how cultural reuse values have been reproduced remains to be studied.

A significant relationship exists between the locations of warehouses and their spatial structure. A GIS-aided process was applied to the selection decision [42]. Significant differences may exist in the spatial scope of warehouse activity, even within the same urban area [43]. The investigation of the relationship between warehouses and their spatial structure enables the comprehension of the former’s role in the evolvement of the urban fabric.

AR and heritage should be integrated. The architectural adaptation of a unique historical form contributes to its integration into the urban fabric. Industrial and warehouse buildings’ heritage should be carefully documented [44]. The overlap of different moments in time stimulates social interaction [45]. While industrial heritages transform our cultural identity from the past to the future [46], heritage forms a link in the creation and renovation of the urban fabric. Visualizing the influence on local development should reconnect the tangible and intangible cultural landscapes of warehouses. As an extension of 3D modeling, 3D scanning, and photogrammetry modeling [47,48,49], AR and GIS should be mutually dependent.

2. Materials and Methods

GIS and AR systems were applied to organize spatial structures using a historical map in QGIS^® (v. 3.22), and the inspection of warehouse details was carried out with the Augment^® (v. 5.6.1+30711) platform by using a smartphone. The building components under study included red bricks, buttresses, corners, openings, decorations, downspouts, and wall finishes (Figure 2).

The entire process started with capturing field images via smartphone AR and involved a series of steps (Figure 3).

• Field image taking (Figure 4a) and 3D model creation: Field images were converted to 3D models using photogrammetry technology in Zephyr^® (v. 7.531).
• AR model uploading and conversion: Previously created 3D models were uploaded to the cloud AR platform, Augment^®, and converted and edited to obtain AR models via a specific QR code link (Figure 4b,c).
• Smartphone AR: A smartphone camera was used to scan the QR code to download the corresponding AR models. Multiple models can be downloaded sequentially (Figure 4d–i).
• AR spreadsheet: A spreadsheet was made based on the details’ latitude, longitude, and QR code links (Figure 4j). The spreadsheet was saved in .csv format and imported into QGIS^® to be displayed on maps (Figure 4k). For each map, the link to the corresponding QR code was displayed.
• Integrating GIS and AR: Each map and associated QR code link were published by applying QGIS2Web (Figure 4j). The maps in QGIS^® (Figure 1) elucidate where the warehouse was located in relation to the center of the targeted production area. Our inspection referred to the map resources provided by the RCHSS Sinica Historical Maps of Taiwan webpages, which host maps from over the last hundred years [1]. After compiling the available resources, maps were selected under a similar chronological distribution in order to compare the changes at relatively comparable time periods in QGIS^® (Figure 1).
  The map and spreadsheet data were converted to a webpage that included a link to the cloud database of the AR model (Figure 4k). Upon clicking the link, a corresponding AR model appears (Figure 4b). Upon clicking the red button on the bottom of the screen, a QR code appears, which can be scanned using a smartphone to facilitate an AR simulation.
• Field access: Once the AR model had been downloaded and displayed as illustrated in “Smartphone AR” (Figure 4d–i), the user could move, re-scale, and rotate the model based on the background context (Figure 4l) for inspection.

The second reconstruction was made by referring to the first one. Both applied photogrammetric modeling to create 3D models of the details. The first one created 3D models from field images of the real physical brick details. The second one created 3D models from (1) two AR models or (2) one AR model and one real physical brick detail. Though both were already in the computer, a secondary reconstruction was needed to document the result after shuffling processes such as scaling or the translation of the AR model to the field. Pictures were taken from different angles centered by the two details, and we applied the photogrammetric model again.

Regarding the field AR model and physical model, the latter proved very useful because its use meant that we did not have to walk back and forth between two distantly located details. In the application, the QR codes placed the target AR model next to the one to be compared.

3. Results

The warehouse, which symbolized the urban fabric configuration, played an active role in defining the old center of development. The warehouse’s specific geographic characteristics reshaped its active role in the industrial landscape.

In this study, GIS technology and AR were used to connect the evolving fabric and the details of the brick warehouse under study, leading to results facilitated by the use of a verification matrix, the integration of AR and reality, rich representation in AR, and the secondary reconstruction of 3D models.

3.1. Verification Matrix of Building Details and Vocabularies

In total, 60 details were created. The comparison and inspection were conducted based on a matrix of details considering field images, a 3D model, and a plain model (Figure 5, Appendix A). The matrix shows the physical details in terms of fidelity in texture and structural details. A plain model was also constructed to reveal the complexity of the 3D details underneath the textures.

The study process was documented by taking images or by taking screenshots on the smartphone. There are two types of screenshots: screenshots with the app menu attached and those without the app menu. The window lintel featured different finish types. The brick layout also showed different spacing for walls situated between two or more windows. The half-extruded brick layout added more interesting detail than the plain ones on the corners of other warehouses.

3.2. Integration of AR and Reality

The downloaded model was superimposed on the physical details on site or in our laboratory for inspection and comparison (Figure 6). The warehouses were renovated not only in terms of their envelopes, but also their plumbing and utilities (Figure 6a). The strip wood pavement and light gray washed floor finish created a juxtaposition outside the warehouse (Figure 6a,b). Old cracks appeared on the walls at both corners of the window (Figure 6d). Ventilation windows were above the entrance, and there was a sealed opening (Figure 6f).

The comparison and inspection were concluded based on a series of transition, rotation, and scaling processes. Adjustments to the openings (Figure 6e,f) were made to differentiate the changes made to the details built in roughly the same decade or with different construction methods using stones from real sites overseas (Figure 6g,h) for curved or flat fences.

3.3. Building Models

Building models were created to provide references for the details. The narrow spaces between the warehouses significantly restricted our ability to take sufficiently clear field images. Eventually, facades were created using additional images taken back and forth along the narrow lanes, with gables also being integrated (Figure 7). Roofs were not included due to the interference of plants.

3.4. Secondary Reconstruction of 3D Models

Field screenshots or paired comparisons were further photogrammetrically reconstructed into secondary-level 3D models. The models were reconstructed for a second time into 3D-printed models as proof of photogrammetric documentation. Two types of secondary reconstructions were made: (1) an AR model and field scene (Figure 8) and (2) an AR model and physical 3D-printed model (Figure 9). The former documented 3D-printed model was also very useful because it could be directly integrated into the AR model for visual inspection.

4. Discussion

The former active, supportive role of warehouses was reformed into a passive role due to the active approach of urban instrumentation by building areas and traffic infrastructure. When the focus of production and storage was switched, the transformed functionality evidenced the degenerated value of the land, while the role was only shared by the former industry culture. The transferred functionality and roles allowed for the determining building details to be documented and inspected.

A warehouse becomes a perfect symbol, since it is directly linked to a local area’s development, from agriculture to industry. If the viewpoint was switched to the fabric that took advantage of the land, the fabric in the surrounding area would also reveal a scenario in which the role of the directly related artifacts had been modified under a more specific interpretation of evolution or degeneration.

4.1. GIS and AR

The evolving fabric also created rich building vocabularies for warehouses. The combination of GIS-aided technology and AR led to a management effort involving the use of webpages. Although AR has the advantage of being able to interact with reality, it falls short of achieving a macro-scale connection with the scale of the urban fabric. QGIS has filled the gap not only with the most updated fabric but also with the historical evolvement of peripherals. It can be found that in 1930, the same location was considered under development and had no structures, while by 1983, the location had an over-developed fabric (Figure 1). Moreover, the AR model’s capacity for interaction can highlight concerns regarding the significant development and transformation of land for alternative use and promote awareness of the importance of preserving cultural heritage by preventing it from being demolished.

4.2. The Use of Smartphone-Based AR in Video Conferences

Smartphone-based AR was used in a video conference during the simulation process to confirm the efficacy of the construction system. For this efficacy evaluation, the setup included a capturing device connecting the smartphone and notebook to transfer screenshots to reconstruct models. Both sides communicated details pertaining to structural support, drainage, and platforms, with the elevation shifting when indoors or outdoors. We found that the employed design created an abrupt interface between the building enclosure and peripheral ground (Figure 10).

4.3. Rich Representation in AR

Smartphones support the representation of models with rich and abstracted textures (Figure 11). In contrast to previous studies, by adopting a photogrammetry-to-AR approach, this study diversified features by changing the level of transparency for an abstracted wireframe appearance. The tested model can be viewed from different orientations to facilitate the inspection of brick edge and layout.

4.4. Second Reconstruction of 3D Models

Furthermore, 3D photogrammetry and 3D printing were used to generate and verify the 60 documented architecture details. An AR model and field background was reconstructed in the same manner for the two aforementioned AR models. This reconstruction was very important for documenting as-built reality in heritage preservation. Moreover, we recursively measured the reconstruction according to the field setting or in the laboratory in a virtual manner. The result was verified by checking the alignment with the original model (Figure 12). As a result, the reconstruction contributed a consistent interface for documentation that is better than remote collaboration or 3D remote diagnosis of complex equipment.

4.5. Contributions and Limitations

This study contributes to historical preservation efforts by presenting a constructive combination of AR analysis, GIS historical map management, video conferencing, and the secondary reconstruction of results. The advantages of this study include the fact that it can be used as a reference for composing AR models, confirming the relative scale of building components, communication between remote sites, and recursive 3D reconstruction.

Our validation experiment also included an examination of whether the proposed system can be applied overseas. In addition to the Taipei area, the most remote site was in Okinawa (Figure 6g,h), Japan, where AR models were downloaded to compare how the local stone layout was different from a former brick construction in Taiwan. The setup included a Sony Xperia 1 II^®, with CHT, Chunghwa Telecom Co., Ltd., Taipei City, Taiwan, which was the international roaming service provider. This smartphone could still facilitate AR interaction 600 km away from where it was originally tested.

We found that the inserted AR model and the background can be reconstructed as a new 3D result. This approach required a novel modeling process to reconstruct the remote interaction result again in 3D after the interaction. As indicated in Figure 13, both sides of the remote parties can either take screenshots from different angles of the AR scene or record the conference video and retrieve frame-by-frame data for photogrammetric modeling.

We found that a reconstructed result can be referred for a follow-up reconstruction process to verify building types, evolving stages, arrangements, and the levels of remodeling required to meet today’s preservation needs. Cross-referencing back and forth between real and virtual models can be carried out to facilitate, for example, ground-level treatment. It appears that the brick details had evolved, having diversified forms and decorations in articulated building parts. As a result, remote manipulation can generate 3D models in either virtual or 3D-printed forms afterward. This is proof of the fact that the relative allocation of the AR model is related to the background. In addition to comparing the two sets of brick details, such models’ application can be extended to virtual installation to confirm results. Moreover, secondary reconstructed results can be cloud-accessed on site to confirm former installation works.

The proposed process is still evolving. The process was recreated recently using Line^® (v. 4.0.0) instead of Skype^® (v. 8.125.0.201). Although both are instant communication applications, the vast popularity of the former led us to apply AR to it for daily use. The first test led to results that were unsatisfactory compared to our experiments using Skype^®. The quality of field secondary reconstruction may suffer under unexpected lighting conditions, when there are internet band width issues, and in cases where video conferencing software is being used on a smartphone with a middle-grade CPU. In our test, video frame lags occurred. Hence, an alternative method involving the use of a screen recording app to reduce the dependence of continuous network connectivity was adopted.

5. Conclusions

The significantly evolved fabric created the unique cultural identity of the former Camphor Refinery Workshop Warehouse. From a macro–micro urban landscape perspective, brick construction, and the related application of material, represents a systematic process that contributes to making connections to adjacent parts. The complexity of GIS led to a new approach that can be used to apply interactive AR to inspect parts with photorealistic textures.

The advantages of the AR-enabled interface include the fact that it assists the composition of AR models, the confirmation of the relative scale of building components, the communication between remote sites, and recursive 3D reconstruction. The results of this study showed that the manipulation of building materials was closely related to the comprehension of traditional industry construction. With a close geographic relation to an old transportation system, the distribution of architectural brick heritage depicted the historical context of a specific evolvement.

This work presented a warehouse-centered AR-based study of evolvement in construction details. Future research should focus on expanding maps with AR as a platform to integrate more details of open cultural landscape systems.