MR-Based Human–Robot Collaboration for Iterative Design and Construction Method

Abstract

The current building industry is facing challenges of labor shortages and labor-intensive practices. Effectively collaborating with robots will be crucial for industry upgrading. This research introduces a MR iterative design and robot-assisted construction mode based on human–robot collaboration, facilitating an integrated process innovation from design to construction. The development of the ROCOS (Robot Collaboration System) comprises three key aspects: (1) Layout Stage: using MR technology to layout the site, forming a full-scale integrated virtual and physical digital twin design environment. (2) Design Stage: conducting virtual iterative design in the digital twin environment and automatically simulating assembly processes. (3) Assembly Stage: translating simulated results into assembly path commands and driving a robotic arm to perform actual assembly. In the end, this research setup two experiments to examine the feasibility of this iterative design–construction loop script. The results showed that although the presence of obstacles reduced the designer’s freedom and increased the number of steps, the designer could still finish both tasks. This means that the ROCOS has value in the prototype of human–robot collaboration. In addition, some valuable findings from users’ feedback showed that potential improvements can be addressed in operability, customization, and real construction scenarios.

1. Introduction

The Architecture, Engineering, and Construction (AEC) industry is undergoing a significant shift from labor-intensive methods to automation based on the revolution of digital transformation [1]. The topic of digital transformation in the AEC industry is rather complex. This research further defines its scope to explore “Smart Construction”, which refers to how to generate intelligent processes from design to construction, even leading to innovation in construction methods. The innovations driven by Smart Construction have three themes: (1) mass production and customization, (2) automation and collaboration, and (3) intelligence and smartness [2,3].

Firstly, under the issue of mass production and customization, the construction industry has long been developing prefabrication systems. Through off-site manufacturing, this approach can reduce construction time and costs, enhance quality control, improve safety, and decrease waste. However, one of the primary reasons why prefabrication methods have not been widely adopted is the overly standardized nature of mass-produced manufacturing. This limitation means that prefabricated building components are mostly used for more standardized building forms, such as factories, making it challenging to accommodate the complex design variations often found in architecturally intricate structures, such as residential buildings. Therefore, the first challenge that Smart Construction must address is customization, particularly by developing efficient methods capable of large-scale customization.

The second issue is automation and collaboration. Unlike the manufacturing of industrial products, it is debatable whether the manufacturing and construction aspects of building components can enter an automated mode, similar to standardized industrial products. Particularly, considering the complexity and customization mentioned earlier in architectural types and designs, full automation in manufacturing and construction might not be as attractive in terms of cost-effectiveness. Additionally, in architecture, whether in design or construction, there are many situations composed of complex conditions that require experienced designers or constructors to make judgments. These conditions cannot be directly produced by automated computers or robots. Therefore, the intervention of “human–robot collaboration” is expected to achieve a dynamic collaboration and balance between the precise efficiency of robots and the comprehensive decision-making abilities of humans.

On the third issue, which concerns Intelligence and Smartness, it is important to note that there exists a distinction between the two concepts. For instance, aspects such as correct design spans, accurate material estimation prior to construction, and efficient energy usage during operation represent Intelligence. On the other hand, making judgments during design regarding the relationship between aesthetics and sustainability, understanding and coordinating different trades during construction, and establishing a balance between green resilience and occupants’ well-being during operation represent Smartness. However, Intelligence and Smartness are not in opposition to each other. Especially within the scope of Smart Construction, it is more about the collaborative relationship, where human wisdom is utilized to apply the intelligence provided by robots. This collaboration is crucial for achieving optimal results in construction practices.

In summary, this research identifies the central issue that Smart Construction must address as “how humans can establish new relationships with robots”. With this premise, this research will focus on “human–robot collaboration”, developing and demonstrating a prototype of a collaborative process from design to construction.

2. Literature Review

This research primarily focuses on human–robot collaborative design and construction within immersive mixed-reality environments. Within this framework, the exploration revolves around how immersive technologies can provide an effective design environment, enabling designers to intuitively transition between physical and virtual realms. This enhances the realism of the design and facilitates alignment with the manufacturing and construction processes. Furthermore, beyond the typical pursuit of aesthetic forms in design, it is crucial for designers to consider the feasibility of manufacturing during the design phase. They should envision how the design can be assembled and constructed, fostering an integrated thought process from design to construction.

2.1. Application of Immersive Technology in Spatial Design

Immersive technology in spatial design generally encompasses three constituent technologies: virtual reality (VR), augmented reality (AR), and mixed reality (MR) [4,5]. According to the Reality–Virtuality Continuum depicted by Milgram and Kishino, VR creates an immersive virtual world that simulates spatial experiences close to the real world through binocular vision [6]. AR, on the other hand, overlays computer graphics or 3D objects onto the real world to produce an augmented effect of reality [7]. The fundamental distinction between VR and AR lies in their essence: VR relies entirely on virtual information and objects to simulate or even replace sensory experiences of the real world, while AR focuses on enhancing and overlaying the real world rather than substituting it. Viewed from the perspective of augmentation, both VR and AR share a common concept, that of reality augmentation. However, AR merely overlays, without genuinely integrating with the actual environment. MR, on the other hand, accentuates the interwoven, mixed space between reality and the virtual. Furthermore, MR, through the concept of “spatial depth”, fosters interactive relationships between the virtual and the real.

Viewed from the perspective of human–robot collaboration, MR is particularly adept at providing sufficient real–virtual reference information to assist designers in spatial design. MR utilizes sensing technologies from both physical and virtual realms to perceive changes in objects, activities, and spatial configurations. This enables collaborative robots or devices to identify and respond in a more immediate and accurate manner [8,9]. This implies that MR should carry more environmental information or real-time feedback to seamlessly blend the physical and virtual worlds, creating a strong sense of reference and interdependence between the two.

In the realm of MR applications, Microsoft’s HoloLens wearable device provides an exemplary example of typical device configuration and usage. Equipped with depth-sensing cameras capable of detecting the real environment and its depth status, as well as tracking user gestures, the HoloLens enhanced its multidimensional application in human–computer collaboration. Led by Cameron Newnham, the Fologram team developed a project called “Steampunk”, which explored the use of steam-bending timber construction techniques. The project, designed by UCL Bartlett School of Architecture, collaborated with the Fologram team to develop a MR application for timber shaping and assembly. Steampunk aims to create complex curved surfaces using multiple timber strips. In addition to using steam to bend the timber strips into curves, precise spatial positioning of these curved timbers posed a significant challenge. To address this, the Fologram team developed a MR-based assembly method using the HoloLens glasses. This method allowed workers on the manufacturing end to see virtual timber models and corresponding data in the actual space. Consequently, they could progressively bend physical timber strips to match the curvature of the virtual model. Simultaneously, during assembly, the spatial positions of each curved timber were presented in the physical environment. Thus, assembly workers, aided by the mixed-model information, could swiftly clarify the interwoven relationships between the curved timbers and assemble them in the correct positions.

The Swiss Federal Institute of Technology (ETH) and the Technical University of Munich (TUM) jointly developed a system called the “Brick-Laying Assist System” [10] based on MR object detection. This system was developed to assist in the construction of complex curved brick walls. In contrast to traditionally linear brick wall constructions, the walls in this project were designed with surfaces that have non-specific curvature. Therefore, each brick had slight displacements and rotations in both position and angle. To assist workers in more precise construction, the Brick-Laying Assist System provided accurate guidance for the position of each brick through MR. Using on-site cameras, the system overlaid the virtual position of the brick wall onto the real scene. Workers could then compare the position of the virtual bricks displayed on the screen with the actual position of the laid bricks. This allowed them to make more precise adjustments for each brick laid until it aligned with the position and angle of the virtual model, achieving precise construction alignment between the virtual and the real.

Through the two cases mentioned above, we can summarize one of the major characteristics of applying MR to spatial design and construction: the need for an interactive relationship between physical objects and virtual information. For example, in the Steampunk project, when bending the wood, it is achieved through the overlap of virtual models and data with MR view, enabling precise physical fabrication of the curved wood in the space. Similarly, in the Brick-Laying Assist System, the alignment between physical and virtual images allows the physical bricks to be gradually adjusted until they match the virtual layout. Therefore, the biggest difference between MR and AR lies in whether they can utilize spatial depth to create corresponding interactive relationships between physical and virtual elements. This will also be one of the key focuses of this research.

2.2. Human–Robot Collaboration for Manufacture and Construction

Human–robot collaboration (HRC) refers to establishing a dynamic system relationship between humans and robots, enabling them to collaborate and complete tasks in interactive environments [11]. Traditionally, although digital manufacture has reduced the distance between design and manufacturing through advancements and integration in computer-aided design and computer-aided manufacturing (CAD and CAM), it still cannot break free from a more linear workflow, where CAD precedes CAM. However, with the development of robot-assisted manufacturing and sensing technologies, the combination of the two has further opened up a cyclical collaboration mode between design and manufacturing, even construction. This mode allows for more design thinking and changes to be integrated into manufacturing and construction, breaking away from the previous linear relationship, and establishing a dynamic inheritance and iterative relationship between design and construction. Willis describes this as Interactive Manufacturing [12], with one of its key mechanisms being the direct manipulability of the target object or material to be manufactured (direct manipulation) [13].

Under the premise of interactive manufacturing, the RoMA project proposes the use of AR for interactive 3D-printing applications between humans and a robotic arm [14]. In this project, the designer uses AR technology to draw virtual local wireframe models on the platform using handles, and the robotic arm equipped with 3D-printing equipment will immediately print the model. Interestingly, the designer can interrupt the robotic arm’s printing at any time, continue modifying or adding virtual model frameworks on the existing physical model, and then continue printing according to the new model. The FormFab project also employs a similar real-time interactive collaboration mode, using blow molding to process plastic sheets [15]. However, unlike traditional blow molding, which usually forms objects as a whole, FormFab’s ability to locally blow mold allows for more flexible and varied shaping. The principle is that the designer wears a glove equipped with sensors and performs gestures such as pulling or pushing in the area where local processing is desired. When the system recognizes the command, it activates a robotic arm with a heat gun to locally heat the area and instructs the blow molding machine to blow air in or retract it, causing the area to expand outward or contract inward, thereby achieving the desired local shaping by the designer.

As human–robot collaboration advances to the level of construction, understanding the dynamically changing relationship between space and objects due to construction will be one of the key focuses of development. In this context, the application of Building Information Modeling (BIM) and spatial perception technology can provide effective spatial references between virtual objects and real spaces. Spatial references can be further interpreted in a digital twin context established by the BIM [16]. This digital twin context is dynamically updated by various sensing technologies, aiming to enable effective convergence of information provided by virtual systems with real machines or even humans, maintaining consistency between virtual and physical worlds. This allows for real-time simulation of ongoing or impending events, providing more effective algorithmic predictions and feedback for construction in the physical world.

Brugnaro and his team proposed an interactive construction system capable of autonomously weaving wooden strips, called Robotic Softness [17]. This system mimics the behavior of birds weaving nests, achieving weaving between wooden strips through automatic path computation by robotic arms. Throughout the process, the system continuously captures real-time weaving patterns using a depth camera placed in front of the robotic arm, determining the path for the next wooden strip to weave. This capability is primarily due to the system’s adaptive computing capability [18]. Adaptive computing is a concept opposed to closed programming. In conventional robotic manufacturing, paths for manufacturing are typically pre-set, such as in wafer fabs for repetitive processing or in 3D printing for printing modeled objects. However, in the weaving construction process of Robotic Softness, because the weaving structure changes locally or globally with the addition of each new wooden strip, once control over the current situation is lost, subsequent wooden strips may have difficulty correctly interweaving into the existing structure. Therefore, the project utilizes depth cameras to grasp the spatial context of the weaving structure and, based on this, adaptively compute the target insertion position for the next wooden strip in real time. Consequently, the weaving path can be continuously updated in real time during the weaving process, accurately weaving the wooden strips together to form a stable structure.

Through the above cases, this research concludes that digital manufacturing and construction in human–robot collaboration require several key elements: intuitive operability, spatial referencing, and adaptive computing. The relationship among these can be understood as the correlation established among humans, space, and machines. Humans need to establish design or work patterns through intuitive operations, while machines need to possess the capability of adaptive computing to dynamically generate adjustable manufacturing or construction processes. Between humans and machines, bridges are built through spatial context, which can also be understood as the need for bidirectional convergence and communication through the digital twin environment established by BIM, serving as a platform for the exchange of digital information and physical actions between the virtual and real worlds.

3. MR-Based Human–Robot Collaboration

In the current AEC (Architecture, Engineering, and Construction) industry, driven by technological advancements and facing a labor shortage, a significant digital transformation from labor-intensive to computation-intensive processes is underway. The three critical factors to initiate this digital transformation are: (1) customization, (2) collaboration, and (3) smartness. Under this premise, this research aims to (1) develop an efficient and scalable mass-customization manufacturing model, (2) create a prototype for human–robot collaborative construction methods, and (3) establish an integrated smart workflow from design to construction.

This research proposes an iterative design method based on MR human–robot collaboration, with the aim of developing a circular construction methodology formed through mutual iterations between a designer and a robotic arm from design to build. The ROCOS (Robot Collaboration System) was developed to provide a digital twin platform for the information exchange between human design and robot construction. The ROCOS is divided into three main stages (Figure 1): (1) Layout Stage: using MR technology to layout the site, forming a full-scale integrated virtual and physical digital twin design environment. (2) Design Stage: conducting virtual iterative design in the digital twin environment and automatically simulating assembly processes. (3) Assembly Stage: translating simulated results into assembly path commands and driving robotic arms to perform actual assembly.

1.

Layout Stage

First, the designer will wear a MR helmet and handles to layout the surrounding site in an immersive manner. At this stage, the designer is surveying and mapping the context of the site to facilitate layout and establish a virtual space model that is proportionally scaled to the real space. When this virtual model is fully mapped with the real space through MR, the designer perceives the real space in an immersive visual manner, while the ROCOS communicates with the designer through the spatial parameters carried by the virtual model. This creates what is known as a digital twin design environment (Figure 2).

Layout of the site and creating a digital twin space serves a crucial function. Apart from offering designers an immersive and full-scale design environment, it also helps prevent conflicts with physical space during the iterative design process or in the assembly path of the robotic arm.

This research utilized Grasshopper Python, a parametric software, for the development of layout functionality to ensure the complete alignment of the virtual model with real space. A specific three-dimensional coordinate trigger point was setup, located at the tip of the handle device. During the initial setup, this point was precisely clicked on a reference object in the real world, recording and transmitting its spatial coordinates. After recording the coordinates of three real reference points (ground/ceiling/edge points), the Rectangle 3Pt function in Grasshopper was used to draw a rectangle, serving as the basis for generating the bounding box of the object (mesh surfaces; Figure 3). This allowed for precise calibration between the real and virtual worlds. Through this method, the achieved accuracy of MR layout (the error between the actual arm touch point and the expected point) was empirically validated to be less than ±0.1 mm (official repeat positioning accuracy was ±0.03 mm). In terms of the centimeter-level accuracy specified in construction standards, this error was nearly negligible, demonstrating highly accurate layout and alignment results between the real and virtual worlds.

2.

Design Stage

The goal of the Design Stage is to assist the designer in iterative design on the design end and to verify the reasonable process of collaboration with the robotic arm on the manufacturing end. Therefore, the Design Stage was divided into two simulated processes: (1) model simulation and (2) robot simulation.

(1) Model simulation: In model simulation part, the designer will operate within a MR interface to conduct iterative design and build a design model. The ROCOS interface allows the designer to use a weaving iterative design method with linear components. By adding lines, the designer creates and observes the relationship of multiple line components in the MR environment. Simultaneously, the ROCOS interface provides real-time feedback on the assembly relationships and collision conflicts between components, allowing the designers to preview and exclude overlaps and collisions as they occur. The conflict segments are marked as red lines, and the designer can directly select and delete components to modify these errors. By adding and deleting lines multiple times, the designer consciously approaches their ideal design aesthetic.

The conflict preview function was developed by the Trim with Brep function in Grasshopper. The Trim with Brep function can detect two kinds of models, one is lines created by the designer, the other is objects built in the Layout Stage. It keeps detecting the conflicts between lines and objects when the design is adding line components. Once the line overlays with the object, the conflict part will turn red to remind the designer (Figure 4).

(2) Robot simulation: In the robot simulation part, the main task is to simulate how the model is passed from the designer to the robotic arm for assembly. The simulated assembly actions of the robotic arm are divided into two parts: assembly path and conflict detection. The assembly path refers to the process where the robotic arm needs to pick up the corresponding materials from the material area and reach the designated position for assembly. For this part of the simulation, the first step is to read the position of the target line (Point A) drawn by the designer from the design model, as well as the positions of the material line in the material area (Point B). Generally, we simplified two line positions into two coordinate points with vectors, representing the direction of lines, to determine the posture of the robotic arm when gripping. Once we obtained the coordinates and vector postures of these two areas, we planned the path from the material area point to the target area point based on the principles of inverse kinematics. In order to avoid unnecessary conflicts during movement, we vertically offset the two coordinate points (A and B) upward by about 10 cm to create two new points (A’ and B’) and plan the path as B > B’ > A’ > A (Figure 5, left). This ensured that when entering the assembly path, the chance of conflicts with existing objects was reduced.

On the other hand, throughout the entire simulation process, the ROCOS continually performs conflict detection to ensure that the robotic arm does not encounter unexpected conflicts during assembly. During each simulation process of B > B’ > A’ > A, the paths of B > B’ and A’ > A are executed in LIN (linear movement) mode for vertical lifting and lowering, while B’ > A’ adopts PTP (point-to-point). The former, due to its specifically raised vertical distance, is less prone to conflict with objects, whereas the latter relies on the system’s own calculation of the suitable path from one point to another and is more susceptible to unexpected collisions along the path. Therefore, the ROCOS employs the Bounding Box feature to detect whether there is interference between objects during the simulation process. Once a conflict is detected, it is highlighted in red to alert the designer (Figure 5, right). The designer can choose to simulate a new path until no conflicts occur throughout the process. Upon validating the outcome of path simulation, the designer can confirm and pass it to the Assembly Stage for real robot-assisted assembly.

3.

Assembly Stage

Once the design from the previous stage is confirmed through simulation, the ROCOS extracts the building information from components, including component lengths, spatial positions, distribution relationships, etc. This information is then outputted as actual path control commands for the robotic arm, generating an RL (Robot Language) file that is transferred to the control case of the robotic arm for actual operation control. The ROCOS system uses the KUKA|PRC plugin developed by KUKA for the Grasshopper parametric design software to generate an RL file. This plugin offers multiple function components for scripting robotic arm control processes and allows for the quick simulation of the robotic arm’s visualized motion by connecting the relevant function components. However, since the robotic arm model used in this research was the HIWIN RA620-1739, not a KUKA product, several key functional modules needed to be rewritten to support the six-axis control of the HIWIN RA620-1739 (Figure 6).

By combining the self-developed RA620-1739 function components (Figure 7, left), the ROCOS can automatically convert the visually simulated paths into HRL (HIWIN Robot Language) files (Figure 7, right) and drive the HIWIN robotic arm to execute the corresponding actions.

In terms of the overall process, the Layout Stage only occurs during the initial setup of the site conditions, while the Design Stage and Assembly Stage form a recurring iterative loop. The ideal scenario involves the designer creating designs and simulating virtual assembly paths in an immersive environment using MR (Figure 8, left). Once confirmed, the ROCOS translates them into commands to drive the robotic arm for actual assembly along the simulated paths, completing one iteration loop (Figure 8, right). Subsequently, the designer and the robotic arm engage in a repeated loop of design to assembly until the task is accomplished.

4. Experiment and Discussion

In this research, we created the ROCOS for the collaboration between the designer and the robotic arm. The system allows designers to engage in iterative design within an immersive environment using MR headsets and handles, synchronously assembling their design creations with the robotic arm. In an ideal scenario, the operational logic of this research sets that for every virtual wooden component drawn by the designer in MR, the ROCOS drives the robotic arm to assemble the corresponding actual wooden component into place, forming a loop from design to construction. Therefore, each new design decision made by the designer is based on the clues provided by the wooden structure previously completed by the robotic arm. To examine the feasibility and applicability of this iterative design–construction loop script, this research further established an experimental environment and conducted operations with experimental and control groups.

4.1. The Experiment of Human–Robot Collaboration for Iterative Design and Construction

The ROCOS human–robot collaboration system developed in this research is suitable for bottom-up iterative design and construction. Therefore, a new design and construction environment was proposed, utilizing MR to provide designers with an immersive full-scale design interface. This environment allows designers to employ discrete stacking of wooden bars. A smaller-scale design and construction area (90 cm × 90 cm × 90 cm) was used as the validation area. For the construction part, the HIWIN RA620-1739 robot arm was positioned adjacent to the design and construction area, with an operational range extending to a radius of 1261 cm. This range effectively covered the entire design area for assembly construction (Figure 9, left).

In the above-mentioned environment, this research setup two experiments (Figure 9, right). One was the control group, where the design and construction area included a 60 cm × 60 cm × 60 cm tabletop as the operational area for placing wooden components. There were no pre-set obstacles above the operational area, aiming to test the designer’s ability to freely operate iterative design within this space. The other was the experimental group, where, in contrast to the control group, the experimental group had several virtual obstacles pre-set in the operational area. The purpose was to test whether the designer can smoothly operate iterative design in the presence of obstacles.

1.

Control group

The control group’s operational area was obstacle-free, and the task was to require designers to connect from the red spot to the green spot, and then to the blue spot, in the most economical way possible. From Figure 10, it can be observed that the designer used 12 wooden bars to complete this task. During the process, the designer had less need for redesigning. In the simulation of robotic arm assembly, there were also fewer occurrences of assembly path conflicts with other objects. Ultimately, the robotic arm successfully assisted the designer in completing the construction with 12 assembly paths.

2.

Experimental group

In the experimental group, the operational area was pre-set with four virtual obstacles, which the designer could see through the MR headset. The task was also to connect from the red spot to the green spot, and then to the blue spot, in the most economical way possible, but without colliding with any obstacles during the process. From Figure 11, it can be observed that the designer used 21 wooden bars to complete this task. Throughout the process, the designer underwent multiple attempts for almost every bar, some due to collisions between the wooden bars placed by the designer and the obstacles, and others occurring during the robotic arm path simulation, colliding with existing wooden bars or obstacles. Ultimately, the robotic arm assisted the designers in completing the construction with 21 assembly paths.

The main difference between the two experiments lies in whether obstacles were present in the operating area. The results showed that the control group (without obstacles) completed the task in 12 steps, while the experimental group (with obstacles) required 21 steps. Although we cannot establish an absolute relationship between the number of obstacles and the required steps, it can be reasonably inferred that the presence of obstacles reduced the designer’s freedom to a certain extent and increased the number of steps. Moreover, to avoid obstacles, the designer needed to go through more trial and error to arrange the relationships between wooden bars. For example, comparing the outcomes of Figure 10 and Figure 11, the latter, due to obstacle avoidance, exhibited more wooden bar configurations aligned with certain narrow spaces, thus affecting the density of wooden bar distribution.

From the perspective of robotic arm construction, although both the control and experimental groups successfully simulated the correct assembly steps and completed the assembly, the experimental group not only generated more assembly steps due to the higher number of steps planned by the designer but also experienced multiple collision errors during simulation for each assembly step. There were two presumed causes for this. Firstly, the addition of virtual obstacles means that if the simulation path of the robotic arm encounters a virtual obstacle, it is considered a failure, leading to an increase in the number of re-simulations. Secondly, as mentioned earlier, the increased density of wooden bar configurations, especially in the vertical direction, easily leads to interference with existing wooden bars along the simulation path. Although the study predicted this issue early on and adopted a path planning method (Figure 11, left), attempting to minimize conflicts by emphasizing vertical entry and exit, there may still be a higher frequency of conflicts when there are too many vertical wooden bars, mostly involving conflicts between the robotic arm’s body and existing wooden bars. The current solution involves repeatedly adjusting the posture of the robotic arm gripper through trial and error, attempting to avoid obstacles in different orientations. However, there may still be situations where no effective simulation solution can be found, in which case modifications must be made from the design side. In the future, it is hoped that machine-learning-based simulations can be introduced to increase the flexibility of grip posture and node position, enabling increased variability in path planning from the construction side, without compromising design flexibility.

4.2. The Discussion of User Feedback

The ROCOS developed in this research underwent feasibility verification through two experiments (control group and experimental group). The basic setup conditions for both experiments were similar, with the main difference being the addition of several virtual obstacles in the experimental group compared to the control group. This was carried out to observe the differences in design operations and outcomes between the two experiments. Through these two experiments, three discussion-worthy topics were identified in this research: (1) system operability, (2) improvement of customization, and (3) potential application in real construction scenarios.

1.

System operability

This research validated the feasibility of the ROCOS through implementation in both control and experimental groups, while also gathering feedback through observational methods and open-ended interviews. Firstly, both experiments successfully completed the designated design to construction tasks, thus preliminarily demonstrating the feasibility of the ROCOS in providing MR design interfaces and human–robot collaborative construction. Users provided feedback that being able to simultaneously observe virtual design and actual construction spaces through MR was crucial, aiding them in effectively examining spatial relationships during design. Especially, the virtual obstacles set in the experimental group were difficult to imagine and grasp spatially during design. However, overlaying them in space through MR allowed precise avoidance during design, serving as a helpful decision-making aid. Furthermore, users found the stacking of lines for design to be engaging and intuitive, particularly with the automatic red marking when lines exceeded construction limits or collided with obstacles, facilitating design adjustments. However, users noted that relying solely on the line-drawing for design could be monotonous. Although the resulting woven designs varied greatly, they might still belong to the same construction method, potentially limiting design imagination and diversity due to method constraints.

2.

Improvement of customization

The current stage of robot construction developed in this research achieved automated assembly functionality. Through digital twin technology, the ROCOS tracked the positions of wooden bars placed by the designer in MR and calculated suitable assembly paths, automatically gripping physical wooden bars for assembly. However, it should be noted that this construction process is not fully automated on the manufacturing side. Currently, the experiments can only use pre-manufactured, standardized wooden bars of the same length for assembly, without the ability to automatically cut them according to the lengths designed by the designer. Therefore, the designer in MR can only stack with a fixed wooden bar scale, reducing the design freedom and resulting in less intuitive operation logic. This research recommends the development of a mechanism for customizing the length of wooden bars. The method could involve using the digital twin model drawn by the designer to provide manufacturing data. Based on the length parameters provided by each virtual line, automated machinery provided by third parties could be used for pre-cutting customized-length wooden bars. This would allow the designer to have more freedom during the iterative design process.

3.

Potential application in real construction scenarios

For the construction industry, the issue of labor shortage is a significant challenge that emerges as the industry matures. Human–robot collaboration not only fills the gap in labor but also serves as a crucial means for the construction industry to transition from labor-intensive to computation-intensive construction. The ROCOS developed in this research demonstrated a design methodology for human–robot collaboration using MR interfaces. Through the information exchange facilitated by digital twin models, it further showcased how front-end design can seamlessly integrate with back-end construction through the automation of a robotic arm, forming a complete loop from design to construction.

We compared our proposed MR interface-based human–robot collaborative construction method with relevant cases mentioned in the literature review. In the aspect of immersive technology in spatial design, compared to the “Steampunk” steam-bent wood technique developed by Cameron Newnham’s Fologram team—which focuses on assisting the positioning and assembly of curved wood in space using an MR environment—our approach allowed designers to directly position virtual wooden strips in space, with robotic arms then placing and assembling the actual wooden strips. By leveraging a digital twin model, the division of labor between humans and robots in design and construction was greatly improved, surpassing the Steampunk method, which still requires human assembly. This represents a significant breakthrough in using MR and digital twin models to enhance human–robot collaboration through spatial computing capabilities.

From the perspective of the building construction method, compared to cases such as RoMA and FormFab—which also use robotic arms for manufacturing, with the former utilizing plastic 3D printing and the latter using blow molding to process plastic sheets—our approach more practically employed robotic arms to construct woven wooden structures. Therefore, the construction method focused more on materials that can produce actual structural effects and was more closely aligned with practical applications in the AEC industry. It has the potential to change the process from design to construction, especially when applied to the steel mesh walls or rebar cages. However, there are several areas that still need improvement before practical application on construction sites. Firstly, there is a need for a change in building materials. Currently, this research employed wooden bars, but from a structural standpoint, a more ideal material would be using rebar for replacement. Continuing with the change in rebar material, aside from improving the structural strength, the joining method at rebar intersections would further transition from hot-melt adhesive to welding. At this point, adding a welding robotic arm to collaborate with the assembly robotic arm would allow for immediate welding fixation at the intersection when the rebar is placed correctly. Therefore, further advancements in multi-robot collaboration will be one of the key areas of future development.

5. Conclusions

This research introduced a MR iterative design and robot-assisted construction mode based on human–robot collaboration, facilitating an integrated process innovation from design to construction. The development of ROCOS (Robot Collaboration System) comprised three key aspects: (1) Layout Stage: using MR technology to layout the site, forming a full-scale integrated virtual and physical digital twin design environment. (2) Design Stage: conducting virtual iterative design in the digital twin environment and automatically simulating assembly processes. (3) Assembly Stage: translating simulated results into assembly path commands and driving a robotic arm to perform actual assembly. The contributions of this research included three main points:

• Establishing an integrated iterative design process from design to construction. This research defined the term “Smart Construction”, which referred to creating an iterative loop process from design to construction. The process from design to construction is no longer a linear sequence but an interactive iterative process between designers and robots.
• Creating an MR-based human–robot collaboration interface and the ROCOS. This research developed an MR-based design interface, allowing a designer to conduct full-scale iterative design in an immersive environment. Through the ROCOS, a human–robot collaboration platform was formed, enabling the designs created by a designer to be synchronously constructed by a robotic arm. This further validated the integrated human–robot collaborative workflow from design to construction.
• Demonstrating an intuitive weaving-based iterative design method and robotic construction mode. Finally, this research invented an intuitive weaving-based construction method for the human–robot collaboration process. A designer can use the MR interface to stack virtual wooden strips to create the desired design shapes. The relatively simple logic of stacking wooden strips also allowed the robotic arm to practically assemble physical wooden strips, thereby participating in the collaborative construction process and demonstrating an effective human–robot collaborative construction model.

This paper also verified the practicality of the human–robot collaboration system, ROCOS, through experiments and user evaluations. We setup the control group (without obstacles) and the experimental group (with obstacles) experiments to examine the feasibility and applicability of this iterative design–construction loop script. The results showed that although the presence of obstacles reduced the designer’s freedom and increased the number of steps, the designer could still finish both tasks. This means that the ROCOS has value in the prototype of human–robot collaboration. In addition, some valuable findings from users’ feedback showed that potential improvements can be addressed in operability, customization, and real construction scenarios. In the future, our ROCOS could be improved toward more diverse and complex workflows, especially by expanding the range of elements available for shape design, thereby surpassing the current basic form of line weaving. The newly added design elements should also allow for parameter adjustments, corresponding to automated manufacturing sizes or shape adaptations, enabling a more customized design-to-construction process. To achieve these improvements, a complex ecosystem formed by multi-robot collaboration will be a crucial direction for advancement.