Intelligent Construction and Management of Landscapes through Building Information Modeling and Mixed Reality

Abstract

Intelligent construction and management processes are essential to ensure the efficiency of planting and maintenance of landscapes in large-scale projects. Unfortunately, various challenges still exist for achieving such a vision. For instance, creating detailed models of various plants to precisely capture the spatiotemporal changes is challenging. Moreover, transmitting massive plant-related data in time for project coordination and remote maintenance is also difficult through traditional landscape management practices. Integrated use of building information modeling (BIM) and mixed reality (MR) techniques could help establish digital representations of various plants with semantic information. Such integration of BIM and MR offers instinctual interactions by blending the physical and digital worlds. This study proposed an intelligent construction and management framework through BIM and MR to (1) establish detailed models of various plants, (2) capture spatiotemporal changes of plants across four seasons for model updating, and (3) establish a real-time data transmission method for effective project coordination and remote maintenance of various plants. The authors used the Jinhu Park project in the Xiong’an New District of China as a case study to validate the proposed method. Results show that the proposed method could support the sustainable development of landscapes in real practice.

1. Introduction

Typical landscapes usually require adequate planting and maintenance (P&M) of a huge number of plants with various types. Construction and management of such a huge landscape area requires extensive manpower and experienced staff to plant and maintain all plants within the landscape. Existing landscape engineers and management staff do not yet have profound knowledge and experience for adequate P&M during construction and maintenance stages of landscapes. Human errors occur frequently and cause inappropriate P&M of plants [1]. Moreover, timely capturing detailed spatiotemporal changes of various plants within the landscape is difficult and time-consuming due to the changing environment of the four seasons [2]. The way in which to achieve intelligent construction and management during tedious P&M processes of a huge number of plants is essential to ensure the sustainable development of landscapes. Creating precise digital representations of various plants and achieving real-time collaborations using building information modeling (BIM) techniques has thus become factorable by landscape architects and engineers [3,4]. To achieve such a vision, it is therefore necessary to (1) establish detailed models of various plants for virtual construction, (2) capture spatiotemporal changes of plants across four seasons for model updating, (3) allow real-time data transmission for effective project coordination, and (4) achieve remote maintenance of various on-site plants for intelligent construction and management of landscapes.

Unfortunately, current landscape management practices do not yet have a systematic framework for intelligent construction and management of landscapes for ensuring the sustainable development of landscapes. This study proposed a framework based on the BIM and MR for intelligent construction and management during tedious P&M processes of landscapes. The proposed framework includes two parts, (1) to establish a parameterized method for detailed plant model creation based on CAD, Excel, Dynamo, and Revit, and (2) to establish a collaborative management method for plant maintenance through the BIM model and MR device. The rest of this article is organized as follows: The second section contains a systematic literature review of BIM and MR technology application in landscape (Section 2). The third section introduces the proposed framework for construction and management of landscapes and provides detailed descriptions to tackle identified practical problems (Section 3). The fourth section verifies the feasibility of the framework through a case study of real landscape engineering (Section 4). The fifth section concludes and provides future research directions (Section 5).

2. Literature Review

2.1. Building Information Modeling (BIM) in Landscape

Previous studies have explored numerous methods for digitizing the P&M processes of landscapes. Cai pointed out the importance of integrating emerging information technologies such as BIM for effective landscape management [5]. Sun and Li have examined the use of BIM in data sharing and cost estimation for proving the necessity of implementing BIM in the landscape [6]. Wu et al. proposed ways to use BIM to solve problems of clash detection, collaborative management, and cost accounting in the landscape industry; analyzed the significance of BIM application in the landscape industry; and confirmed its feasibility [7,8]. Buhmann conducted a literature review and revealed the development of BIM in the landscape field within the past 10 years [9]. RÜCKER investigated various ways to convert or link landscape elements into BIM software so that the data imported into Revit was improved for considering brake lines through the integrated development environment Dynamo [10]. In recent years, BIM are favored by engineers and have become widely used in the landscape field. For example, in the Suzhou Guofang Park project, the staff used BIM technology through the method of building a platform to coordinate the design of various disciplines and carry out visual simulation of the model [11]. BIM was used for clash detection and construction simulation when constructing a landscape inside a resort hotel in Guangzhou for achieving cost control [12].

2.2. Mixed Reality (MR) in Landscape

MR technology is developed on the basis of advancements in computer vision, graphical processing, display technologies, input systems, and cloud computing. Such a technology blends physical and digital worlds for providing natural and intuitive interactions between humans, computers, and the environment. In recent years, MR, which blends real and virtual worlds, has been attracting attention as a visualization method for improving landscape management [13,14]. Van Krevelen et al. applied MR-based landscape visualization is achieved by superimposing a three-dimensional computer graphics model based on planning and design onto live images captured by a web camera or similar device [15]. Lou et al. used MR to study landscape design from free viewpoints in real space and represented the environment surrounding the object under consideration by using live images [16,17,18]. Tao explored and applied MR to the construction of space environment in landscape design, expanded landscape vision space through MR, and realized the most reasonable utilization of landscape resources [19]. Hatanaka described a system that uses a landscape overlay display of mixed reality. The system uses MR to overlay a virtual landscape of a remote location on top of the real experience, so as to extend the entertainment experience [20]. Nakabayashi et al. proposed a method for large-scale MR landscape visualization that enables the recognition of physical objects of the same type individually and accomplishes visualization when a three-dimensional virtual model is superimposed between multiple physical objects of the same type [21]. Kido et al. developed an MR-based system for evaluating future landscapes. The developed system could help to evaluate the landscape index through semantic segmentation for comprehending dynamic occlusion processing and landscape index estimation [22].

2.3. Integrated Use of Building Information Modeling (BIM) and Mixed Reality (MR)

Some scholars have explored the comprehensive utilization of BIM and MR [23,24,25,26]. Zhou et al. examined the feasibility of integrating BIM and MR by examining the software and hardware architecture design and different virtual scenes being used in MR technology [23]. Jiang et al. combined BIM with MR and proposed an efficient auxiliary method for mechanical and electrical installation, so as to make up for the lack of visualization of BIM during mechanical and electrical installation [24]. Brito proposed a framework that integrates multisource facilities information, BIM models, and feature-based tracking in an MR-based setting, so as to achieve remote collaboration and visual communication between the fieldworker and the manager at the office [25]. Prabhakaran proposed a novel methodology for the application of MR in design coordination as well as investigating the impact of introducing MR into BIM workflow with a focus on the identification and avoidance of clashes [26].

The literature review efforts show that the current landscape research has not yet involved time-varying plant model creation and field plant visual maintenance based on BIM and MR. The scientific contribution of this study is to establish an innovative framework that combines BIM and MR in the landscape to realize the creation of plant time-varying models and the visual maintenance of field plants. The proposed framework aims at achieving effective management and control of plants during various P&M processes. Thus, this study has certain practical significance for promoting the digital application of landscape and the sustainable development of landscape ecology.

3. Intelligent Construction and Management of Landscapes through Building Information Modeling and Mixed Reality

This study proposed a framework for intelligent construction and management of landscapes for ensuring adequate construction and maintenance of landscapes. The proposed framework contains two parts: (1) virtual construction of plant model, and (2) visual management of plant growth (see Figure 1). We used BIM to establish detailed digital representations of various plants within the landscape. The established digital representations could represent multivariate plant-related construction information (e.g., attributes, position). On the other hand, the proposed method allows users to visualize the real-time plant condition through a MR device. Moreover, the unique characteristics of the time, space, and information expression of various plants could be visualized in the environment combining the virtual world and reality. Thus, the proposed framework could be helpful in improving (1) project coordination and (2) the traceability of plant maintenance information for achieving scientific and effective landscape management.

3.1. Virtual Construction of Plant Model

3.1.1. Refined Full-Attribute Time-Varying Modeling

Creating better plant visualization of the model is important for landscape design and construction. Establishing a fine tree model is thus necessary to enhance the realism of the virtual landscape. Due to the complex interdependence between plants and the environment, modelers should pay more attention to capturing changes in plant appearance and morphology with tree age, seasons, and other factors [27]. Modelers use Revit to finely create the full attribute of the model, create a static plant library, and assign different growth states to plants, so as to realize the time-varying driving of static attributes and dynamic attributes of the plant model. Such efforts could help to promote the periodic update of the model data information and improve the authenticity of the model simulation effect.

Static Model Creation

Revit provides a library of plant families. The staff could (1) add other plant objects to existing families in any new project when needed, (2) modify plant families as appropriate, and (3) create the desired plant family directly according to the actual situation. In the current practice, the staff used Revit to retrieve the plant family library inside and compared it with the plant species needed in the landscape, supplementing the missing plant family. Subsequently, the staff adjusted or recreates the unsuitable plant family so that the plant species required for the actual project were arranged in the plant family library. The coding standard of the BIM plant family library is formulated before the plant family library is created for convenience. Moreover, the corresponding plant model can be directly linked through the coding. At the same time, plants are reasonably classified (see

Table 1. Examples of landscape plant classification.

Level 1	Level 2	Level 3	Level 4
Plant	Trees	Evergreen trees	Spruce
			White fir
			Podocarpus
			…
		Deciduous trees	Poplar
			Weeping willows
			Peach trees
			…
	Shrubs	Evergreen shrubs	Palm
			Gardenia
			Ginseng fruit
			…
		Deciduous shrubs	Peony
			Chinese rose
			…

). In this way, the standard and uniqueness of plant components in the plant library could be realized, which is beneficial to the identification and use of the model in the project.

Dynamic Model Creation

Dynamic changes in various plants should also be considered when creating detailed and precise plant models. Specifically, spatiotemporal changes in leaves across four seasons need to be reflected in the plant model. According to the situation of the project site, we used the parameterized engine tool in Revit to control the spatial geometric parameters and type parameters of plants with models with different size attributes. Hence, different seasonal growth states of each plant can be created. Figure 2 shows a picture of the four seasons of the locust tree and Figure 3 shows the growth picture of the ash tree. Such creation provides additional materials for better visualization of the plant status at the project site in the MR device. It is worth noting that the creation of the dynamic attribute model here does not apply to herb plants that are spring-born and winter-dead or are short-lived.

3.1.2. Parameterized Planting

Plant types in a landscape are usually diverse and large in number. A huge number of plants usually occupy a large area, which makes it difficult to determine the specific location of the model. In the virtual planting process, CAD, Excel, Revit, Dynamo, and other tools are used on the basis of a combination of parameterization to achieve precise positioning, batch creation, and corresponding coding information of the required plant models. Designers could use the program developed based on CAD, Excel, and Dynamo tools for modeling various plants in Revit much more efficiently. Such efforts could not only greatly improve the modeling efficiency but also make the model location more accurate and specific. Hence, precise planting of plant models based on parameterization and full coverage could be achieved.

Coordinates Pick-Up

Designers use the built-in “data extraction” function in AutoCAD software (as shown in Figure 4) to extract the data information (e.g., block attributes, graphic features, positions) of the plant objects in the design drawings into an Excel spreadsheet. Then, the data in the table and the table styles are further processed to pave the way for the precise positioning and batch planting of subsequent models. The designers can organize and optimize the information in the table or format an existing table as a template through a specified table style. When the data change, the extracted data can also be automatically updated to keep pace with the data source. After the designers are satisfied with the form, the data extraction process is completed. If they need to share the extracted information with others in the project, they can output the same extracted data to an external file.

Dynamo-Based Precise Planting

Dynamo is a graphical programming plug-in for custom building information workflow based on Revit [28]. Converting the Excel data exported and processed by CAD into the data in Dynamo in the script. The “Data. Import Excel” node is used to read all plants in the Excel table to realize the precise positioning of the plant model. Finally, the model is transferred to Revit through the “Family Instance. By Point And Level” node, as shown in Figure 5. The batch creation of different types of plant models can be realized. Similarly, the “Select Parameter Type”, “Parameter. Create Project Parameter”, and other nodes are used to create the required parameters in the script. The elements are numbered in the order from north to south and from west to east, and model coordinates are obtained in the order of numbers. At the same time, the coding is integrated, and the coding is given to the model. Finally, the coordinate position of the model is assigned by combining the coding and the obtained coordinates, as shown in Figure 6.

For the later model modification, designers only need to modify and run the corresponding logical parameters in the Dynamo file, and Revit will get the changed model under the drive of Dynamo [29,30]. Using Dynamo visual programming for parametric design can save a lot of time for repeated operations and improve the accuracy of the location of the model.

3.2. Visual Management of Plant Growth

In the maintenance stage of landscape, the application of BIM enables engineers to avoid looking at chaotic, trivial, and two-dimensional data. All data information is presented in an intuitive three-dimensional model, which is conducive to the development of maintenance work. However, the virtual engineering model data displayed by BIM still stays on the two-dimensional display screen, and it is difficult for it to be visually displayed in the real world. The MR device only provides a channel for the synchronous interaction between building information model data and the real world [31].

Applying BIM and MR to the growth process of landscape plants will help to better realize the visualization of landscape spatial information and the contextualization of virtual and real integration, which can provide a more efficient and practical method for the maintenance management of the landscape.

3.2.1. Field Location of the Virtual Model

The display method of model information still stays in the two-dimensional display screen, which is prone to mismatch with the actual situation on the spot. Using a Microsoft Hololens2 holographic glasses (hereinafter referred to as “Hololens2”) device can transform the original two-dimensional display of the office area into a three-dimensional holographic projection on the scene, and then the 1:1 BIM plant model and its data information are placed in the physical space of the park.

The placement point of the BIM model in the development program is selected, and the spatial position of the model is determined through the mapping of the virtual anchor point in the platform and the landmark point in the real scene. The registration and positioning of the specific location of the model are carried out by the positioning function of the Hololens2 device itself, and fine-tuning (rotation, scaling, etc.) is used to register and position the specific position of the model, as shown in Figure 7, making the virtual image merge into the objective world. Meanwhile, the virtual model and the real object should conform to the regularity of near-large and far-small, the consistency of geometric dimensions, and the correct occlusion relationship [32,33], so as to achieve the effect of using the model to accurately simulate the field physical object.

3.2.2. Real-Time Model Data Acquisition

A massive amount of virtual data in the BIM model could hardly be transmitted in real time during the maintenance phase of a project. Using the Hololens2 device enables real-time control of field conditions and model design conditions. Relevant plant information is encapsulated in advance in the BIM model, such as component attribute information (geometric information such as height, canopy diameter, diameter at breast height, branch point, and soil ball size, and other non-geometric information such as plant category, cost, area, planting requirements, and application area) and planting process information (planting team, etc.). By setting the button to extract relevant information in the virtual three-dimensional display platform provided by the Hololens2 device, the information can then be called on-site to provide an accurate and reliable data reference basis for planting in time, so as to improve the transmission efficiency of data information between model and site.

3.2.3. Effective Project Collaboration

Existing applications using BIM could only realize the multi-professional collaborative office in two-dimensional display screens. The intervention of MR technology could help to realize real-time multi-professional collaborative communication in a three-dimensional space. Connecting multiple MR devices under the local area network allows users to view the model in the same virtual space even if all users are not in the same location. At this time, multiple Hololens2 devices have the same anchor point, and the operation of one Hololens2 device can be synchronized to several other devices to realize multi-machine interconnection. Then, the suspicious points in the planting process of the project are communicated and exchanged in real time according to the field planting situation, so as to achieve the purpose of collaborative communication by multi-party designers in three-dimensional space.

3.2.4. Accurate Trace of Maintenance Information

Adding a timely update log function and corresponding editing buttons to the Hololens2 device could effectively support the recording of environmental information (e.g., air humidity and weather conditions on the day of plant maintenance; the specific time, staff, water consumption, types and amounts of fertilizers and pesticides). Meanwhile, editing permissions and log reminders are set up to supervise and urge the staff to regularly maintain, inspect, and accept the plants in the landscape, as well as to increase their attention to plant maintenance. In this way, managers could improve the management of plant maintenance to ensure all plants could be perfectly maintained. Hence, the proposal could help to realize the accurate traceability of plant maintenance information for sustainable landscape management.

3.2.5. Animation Instruction for Field Maintenance

After placing the BIM model of the same proportion and its data information in the physical space with the Hololens2 device, plant maintenance processes can be animated in the glasses by gesture or button. During the demonstration, commands such as play, pause, and replay of the animation can be performed by a gesture. After the completion of each animation, notes and other relevant information about plant maintenance methods are displayed to assist the training of plant maintenance methods and the actual maintenance on site, so as to achieve scientific management of landscape plants.

4. Results and Discussion

The Jinhu Park in Xiong’an New District is currently the largest livelihood project under construction in Xiong’an. The park covers a total area of about 248 hectares with 23,000 trees and 46,000 shrubs [34]. This project integrates BIM and MR for achieving efficient information management of landscapes during the planting and maintenance period. The following sections demonstrate (1) parametric modeling, (2) parametric coding assignment, (3) BIM and MR data interaction, (4) collaborative communication, (5) information tracing, and (6) visual maintenance, as shown in Figure 8.

4.1. Planting PHASE

4.1.1. Parametric Creation of Information Model

In this project, four tools (i.e., CAD, Excel, Revit, and Dynamo) were combined and applied on the basis of parameterization. Then, the required plant models were created in batches according to the specific locations of the models, and the coding information was given to them, which greatly improved the modeling efficiency. In terms of model creation and precise model location, firstly, we enriched the plant types related to the project in the plant family library in Revit and performed refined dynamic and non-dynamic attribute modeling. Secondly, we used CAD to complete the extraction of plant model positioning coordinate data, and the data were processed in Excel. Thirdly, we created a script for parameterized generation of plant models in Dynamo. Finally, we read the processed Excel file in Dynamo. After running, the plant information model can be generated in batches to achieve the effect of virtual construction, as shown in Figure 9.

4.1.2. Parametric Coding Assignment

The authors have adopted the “Ecological Engineering Landscape Plant Model Classification and Coding Standard” for coding. The data were sorted according to the “China Xiong’an Group BIM Standard System Landscape Part (Fourth Part)” to form an Excel file. The staff can simply assign model code information with one click by reading the file and using the parameterized model encoding script produced in Dynamo programming (see Figure 10).

4.2. Maintenance Phase

4.2.1. BIM and MR Data Interaction

In this project, we used the Hololens2 device combined with BIM and MR to achieve effective data sharing and information transmission in the application of landscape. Most of the functional modules involved were developed using the C# language. The development environment is Windows 10. The development tools can use Unity 2019 pro and Microsoft virtual studio community 2019. Moreover, the basic development kit was “Microsoft. Mixed Reality. Toolkit. Unity. Foundation.2.4.0. unity”. The realization of each module and its business logic was considered according to actual needs, and the first element was designed to reduce errors to realize the cooperative combination of various modules.

We parsed, extracted, assembled, and encapsulated the plant model data designed in Revit and exported the model and ODBC database. After importing the model data to the database server, an information management platform based on BIM and MR could be constructed on the basis of the Hololens2 device. We debugged and integrated the platform environment in the visual management database, accurately matching the data, indexing each other, and forming the overall model. Then, the digital display and interaction of the plant information model in the virtual and reality coexisting environment was able to be realized. The specific operation process is shown in Figure 11 below.

The authors transferred the BIM data to the Hololens2 device. Then, the device was used to place the same proportion of BIM models in the physical space. Subsequently, we were able to retrieve relevant plant information (e.g., ID, name, location, height, diameter at breast) at any time on the site for effective management of various plants. Data in the BIM model were visualized to improve the data transmission efficiency between the model and the site, as shown in Figure 12.

4.2.2. Collaborative Communication

In this project, we used BIM and MR for achieving accurate transmission of multi-dimensional model information and the collaborative communication of the model by multiple parties in virtual space. Multiple designers can simultaneously observe the three-dimensional effect of the landscape after the planting of plants in advance and visualize clashes with the park structure and site preset pipeline in the process of plant growth using Hololens2 devices. While protecting the aesthetic effect of landscape design, the conflict of plants is avoided in advance, and the collaborative communication and management of multi-professional models are achieved, which indirectly reduces the later maintenance cost.

4.2.3. Information Tracing

In this project, we implemented the log function into the Hololens2 device. Landscape staff could use such a function to record various critical information during plant maintenance, such as the maintenance time; the name of the person; the use of water, fertilizer, and pesticides; and other related information during plant maintenance. All such information could be archived and updated in time to establish a comprehensive database for plant maintenance. Landscape managers could also use such a function to accurately trace the maintenance work for monitoring the maintenance process and examination of the maintenance quality. In this way, managers could in a timely manner regulate the plants with poor growth conditions and account for relevant staff. This ensures sufficient soil, moisture, and nutrients and provides the information system of plant maintenance.

4.2.4. Visual Maintenance

This project applied BIM and MR technology to field maintenance. The maintenance methods of different plants in different growth stages in the landscape can be encapsulated in text or animation into the corresponding BIM model or MR device in advance. Field landscape staff could use the Hololens2 device to compare the real plants with the virtual models, as shown in Figure 13. Moreover, field landscape staff could also extract specific treatment measures for specific plants to realize required visual maintenance procedures for achieving meticulous and scientific maintenance of plants.

5. Conclusions and Future Works

In the case study, the collaborators of this park project were mainly landscape designers, project managers, and maintenance staff of the landscape. On the basis of the construction management framework proposed in this paper, landscape designers could use the three-dimensional design for intelligent design of landscapes. Project managers could also benefit from the proposed method as it allows them to provide guidance at remote locations. Moreover, maintenance staff could carry out plant maintenance activities much more effectively using the collected maintenance information. In conclusion, the established informationized construction and management system for landscapes was proven to be able to tackle the following challenges in ensuring sustainable landscape ecology:

(1)

The proposed method integrated multiple parametric design tools (e.g., CAD, Excel, Revit, and Dynamo) for the parametric creation of the full-attribute refined model at the precise location, as well as parametric assignment of the corresponding model code. Thus, the cumbersome problem of BIM plant model creation was successfully solved.

(2)

The proposed method encapsulated massive virtual data of the established BIM model into the Hololens2 device for achieving real-time data transmission at the job site. Thus, the problem of the disconnection between the job site and the model data was successfully solved.

(3)

The proposed method achieved a collaborative model design platform by connecting multiple Hololens2 devices. Thus, the problem of conflicts in multi-disciplinary designs was successfully solved.

(4)

The proposed method encapsulated maintenance information of various plant types at different growth stages (e.g., maintenance time, maintenance methods) into the Hololens2 device for achieving effective tracing of the historical maintenance information of the plants on site. Moreover, the information-based scientific maintenance with animation guidance is realized. Thus, the problem of inadequate maintenance of landscape plants was successfully solved.

In summary, the proposed method enables effective model creation and visualization of various plants. All landscape-related personnel could participate in design, construction, and maintenance processes and communicate with field landscape staff in time to provide more scientific and reasonable planting and maintenance practices for landscape management. Thus, the integrated use of BIM and MR for construction and management of landscapes could help to improve the efficiency of (1) collaboration among multiple landscape personnel and (2) plant construction and management. The proposed method has pointed out the direction for the digital combined application of BIM and MR technology in the landscape. Limitations still exist in the proposed method. For example, the proposed framework proposed is mainly used for shrubs, trees, and other perennial woody plants in landscapes. The proposed method might not be suitable for small herbs, vines, ferns, and other similar plants. Further exploration is still necessary if the complete landscape is visually displayed on Microsoft devices.