Advancing Sustainable Construction: Discrete Modular Systems & Robotic Assembly

Abstract

This research explores the SL-Block system within an architecture framework by embracing building modularity, combinatorial design, topological interlocking, machine learning, and tactile sensor-based robotic assembly. The SL-Block, composed of S and L-shaped tetracubes, possesses a unique self-interlocking feature that allows for reversible joining and the creation of various 2D or 3D structures. In architecture modularity, the high degree of reconfigurability and adaptability of the SL-Block system introduces a new element of interest. Unlike modularization strategies that emphasize large-scale volumetric modules or standardized building components, using small-scale generic building blocks provides greater flexibility in maximizing design variations and reusability. Furthermore, the serial repetition and limited connectivity of building elements reduce the efforts required for bespoke manufacturing and automated assembly. In this article, we present our digital design and robotic assembly strategies for developing dry-jointed modular construction with SL-Blocks. Drawing on combinatorics and graph theory, we propose computational design methods that can automatically generate hierarchical SL-Block assemblies from given shapes. To address the physical complexities of contact-rich assembly tasks, we develop robotics using two distinct methods: pre-programmed assembly and sensor-based reinforcement learning. Through a series of demonstrators, we showcase the ability of SL-Blocks not only to reconfigure conventional building tectonics but also to create new building configurations.

1. Introduction

Our society faces the global challenge of serving an enormous demand for housing now and in the coming decades with an industry whose productivity is traditionally low and which suffers from a shortage of skilled workers. At the same time, innovations are needed to dramatically reduce the amount of CO₂ emissions and waste produced by construction sites [1]. These challenges require a fundamental rethinking of material flows, construction techniques [2], the automation of labor-intensive tasks but also the notion of permanence in architecture. Permanence in architecture heavily relies on the antique, Vitruvian idea of firmitas, in which mass and solidity are meant to form an eternal architecture [3]. This notion of permanence drives architectural design, and it formed our design theories [4] and construction systems [5]. Building codes and contracting are geared towards linear building element use. However, buildings face economic obsolescence long before their building systems fail. Such a take–make–waste approach fosters material depletion and waste production. Buildings, which are designed to last for eternity, are prematurely demolished and turned into waste. The urgent need for more reuse in architecture, engineering and construction (AEC) clashes with existing buildings, which are not designed for disassembly.

This research therefore proposes to fundamentally rethink the notion of permanence in architecture by proposing a process of continuous transformation of the built environment. Here, permanence is not established within a single building but through building elements that endure as reused materials in potentially numerous consecutive buildings, sites, and functions. One important question in mastering this challenge concerns how building elements can be designed to maximize the adaptability, allowing for spatial and functional changes in various building types and scales. Industrial practices in modular construction have demonstrated the advantages of using standardized, prefabricated building components and modules, especially in projects such as hotels and residential or office buildings, whose room units are well suited for repetition [6,7,8]. However, moving towards greater degrees of prefabrication from elements (bricks, scantlings, steel rods, etc.) to components (columns, walls, beams, etc.) to container-like volumetric modules forming entire rooms, the level of adaptability and automation in the building system progressively diminish. Elements and components designed for large-scale volumetric modules often lack the flexibility required for use in varying scales or configurations. Furthermore, while the manufacturing and assembly of volumetric modules can be automated in the prefab process, the different types of connection possibilities and on-site assembly may require more complex machinery coding and operations.

Therefore, unlike the modularity in the current building industry, our focus is on a minimum type of building elements and connections for a maximum degree of design variations, differentiation, adaptability and openness [9,10]. In our modular approach, we consider building elements as standardized, discrete and generic building blocks with a common joining principle. These blocks are small-scale, self-interlocking, and can be assembled in a way that allows for a broad variety of design combinations. A cascading hierarchy defines building block aggregations and how they are transformed into large building components (e.g., columns, walls, and floor slabs) or other configurations, responding to architectural requirements for permanent adaptability. In that sense, the SL-Block becomes a central interest in our project due to its high degree of reconfigurability and adaptability. To explore the implications of the SL-Block system in modular architecture, we propose a continuous digital process chain from design, materialization, and pre-assembly to construction.

In this article, we present our work on the prototypical development of a modular building system and its assembly by autonomous robots. Our interdisciplinary team developed elements that can be dry joined into interlocking assemblies. The elements differ in material and function, enabling load transfer, light transmission, etc. The material system is optimized to be assembled, disassembled, and reassembled by autonomous robots. They determine their Task and Motion Planning (TAMP) based on a high-level, abstract description (3D model) and “understand” the weight, dimensions, and surface properties of elements using tactile sensors. Machine learning and sensor technologies help to autonomously perform construction tasks that cannot be programmed in advance due to the need for permanent adaptability.

2. State of the Art

2.1. Digital Modularity in Architecture

This research operates with modules called SL-Blocks that are significantly smaller than building components such as columns, walls, beams, or floor slabs. The SL-Block is an octacube made from a voxelized S and L shape joined together. Developed by Shih [11], two SL-Blocks can be dry joined in six different ways which lead to a large variety of aggregated shapes. Thus, many SL Blocks aggregated in a reversible fashion can form the above-described building components. Once those components are obsolete, they can be disassembled into SL-Blocks to form novel and differently shaped parts.

SL-Blocks can be considered as “digital material”, as defined by Neil Gershenfeld from the MIT Center for Bits and Atoms [12]. He suggests extending the concept of the digital from the computer numerically controlled (CNC) processes of manufacturing to the materials themselves. A “digital material” is not continuous and homogeneous like steel, but consists of discrete repetitive elements that form a large whole through reversible connections [13]. Like LEGO blocks, all elements have self-calibrating geometric properties that precisely control the joint. Unlike continuous 3D printing, the elements can be made of different materials as long as they have a common joining logic. Digital materials thus offer full reversibility and digital programmability of physical processes. When constructing with digital material on an architectural scale [14], bottom–up aggregation of elements is seen as a formative force and not as a subsequent subdivision of the large form. Sanchez and Retsin suggest a digital material approach in architecture [15,16].

Topological interlocking [17] is a design principle by which building blocks of special shape can form a freely spanning, non-monolithic, reversible load-bearing system only through their geometry and the topology of their joint boundaries. Estrin et al. studied multi-layer interlocking elements while exploring the use of hybrid materials [18]. Meanwhile, Mather et al. [19] developed the finite element method (FEM) to investigate the mechanical properties of such systems. Tessmann [20] presented parametrically designed interlocking elements for rebuilding planar and curved surfaces. Weizmann et al. [21] investigated the correlation between block geometry and structural behavior in flat assemblies of convex interlocking blocks. Jonathan Hiller and Hod Lipson [22] investigated spatial lattice packings based on digital material, which are joined via interlocking components. Construction criteria and self-calibrating 3D voxel geometries are presented in their work.

Scientists at Nanyang Technological University and Tel Aviv University developed a mathematically formalized method for designing recursively interlocking 3D puzzles. A specific sequence of assembling or disassembling also ensures interlocking and stability during construction [23]. In 2017, the same team further developed the computational approach for designing reconfigurable interlocking furniture. The developed graph models the solution space of different reconfigurable assemblies with part and joint compatibility across different forms [24].

Dry-joint systems have started to appear in AEC practice. One example is TRIQBRIQ, a wooden brick made from off-cut micro elements (https://triqbriq.de (accessed on 23 July 2024) or the Drystack that replaces mortar in brick walls with vinyl spacers (https://drystack.nl/ (accessed on 23 July 2024)).

2.2. Robotic Assembly in Architecture

Automated assembly is still underdeveloped in architecture compared to other industries. The first research projects on this topic date back to the 1990s [25,26]. Today, there is a small range of commercial projects involving masonry robots such as the SAM100 (Semi-Automated Masonry) (SAM100–Construction Robotics, n.d.) and the Hadrian X ROBOT (Hadrian X^®|Outdoor Construction & Bricklaying Robot by FBR, n.d.). At the same time, there is extensive research interest in the field of masonry automation [27] partly driven by shortage of skilled workers and demographics. Robots are becoming a necessity for the construction industry [28]. Current developments in robotics, machine learning, and sensor technology open up the possibility of researching automation for complex and autonomous construction tasks with new approaches [29].

There is a need for research bridging the gap in the digital process chains of today’s construction processes [30]. Current research explores robot-assisted production and the joining of components made of stone, wood, and steel [31,32,33] and assembly of contact-rich and tolerance-prone, press fit wooden lap joints using Reinforcement Learning [34]. Human–robot interaction [35] for construction processes is explored, as well as the use of robots as a substitute for temporary support structures during assembly [36].

In addition, various teams are researching the development of components that are specially adapted to robot-assisted assembly. Thomas Bock [37] coined the term robot-oriented design (ROD) in 1988. This involves taking robot-related parameters into account in the early design and production phases of components in order to create defined conditions for the use of robots on-site. Rogeau et al. [38] develop element joints for the robotic assembly of wooden panel constructions. The wooden joints were beveled to allow for material tolerances and enable automatic calibration. A building block specifically for robotic assembly was developed for the Acoustic Brick Wall by Gramazio and Kohler [39]. The geometry is determined by a self-correcting “fall-in-place” mechanism to automatically avoid horizontal misalignment. Ariza and Gazit [40] define the element size and joining detail of a mushroom-shaped roof, taking into account the maximum reach of the assembly robot. Using drones to place building blocks, Goessens et al. [41] develop lightweight elements that self-align during assembly.

Another direction in robot-assisted assembly research is the modularization of robots that almost merge with a material system to be assembled. These material robotic systems (MaRS) are mobile robots that can assemble discrete cellular structures [42]. The robot and the material are designed to form a coherent system in which both entities are coordinated. The elements are designed for robotic assembly, and the robots are designed for their assembly so that both sides’ technical and material constraints are considered [13]. The distributed robotic assembly system for in situ timber construction project by Samuel Leder and Ramon Weber [43] uses wooden components temporarily as a robotic element. To achieve the functionality of a robotic arm, mobile joints grip the rods and form a six-axis robotic arm that disassembles and reassembles itself after the component has been assembled. At the Bartlett School of Architecture (UCL), Dafni Katrakalidi, Martha Masli, Mengyu Huang, Man Nguyen, and Wenji Zhang [44] designed a robot in the AssemblerAssemble project that becomes part of the building and remains inseparable from it.

3. Methodology

The prefabricated building can be abstracted as a complex assembly system, where raw materials are manufactured into building elements that are hierarchically assembled into building parts and components, with complexity increasing until a large functional whole is constructed (Figure 1). According to [45,46,47,48,49], complexity is a matter both of the sheer number of distinct elements and the nature of the interdependencies among those elements. To effectively manage building complexity, one possible way is to use a limited number and type of building elements with modularized sizes and standardized connections, as a means to reduce the distinctions and interdependencies. In this context, this work focused on the development of SL-Blocks as modular building elements and investigated their impact on the design and assembly of prefabricated buildings.

At the element level, the utilization of SL-Blocks in design and engineering necessitates a systematic definition for the geometric hierarchies, including geometric composition, combinatorial rules, and aggregations. These foundational principles are introduced in Section 3.1. Moving to the digital design in architecture, we first proposed two computational methods: (i) a grammar-based approach enabling bottom–up design of various SL-block assemblies through string operations (Section 3.2.1); (ii) a constraint-based computation that can automatically break down desired forms into hierarchical assemblies of interlocking SL-blocks (Section 3.2.2). Furthermore, using our developed computational design tools, we explore the design consequence of SL-blocks in modular architecture and discuss the implications of SL-Blocks as building elements (Section 3.3).

To address the complexities involved in robotically assembling SL-blocks, we developed both a high-level controller and a low-level controller for the robots. The high-level controller is responsible for planning the entire sequence of robot movements during assembly. Automated and autonomous methods are, respectively, introduced in Section 3.4.1 and Section 3.4.2. The automated method utilizes manual heuristic search to compute the robotic sequence. In the autonomous method, we use machine learning techniques to empower robots with decision-making capabilities for design and assembly planning. This method enables the robot to independently translate a user-desired shape and the available building blocks into its action sequences. The low-level controller (Section 3.4.3) focuses on handling the intricate details of physical contacts and frictional challenges during the insertion of SL-blocks. To develop this controller, we explored model-free reinforcement learning, tactile sensors, and tracking systems. Our goal is to combine both controllers into a cohesive robotic system to achieve the best possible autonomous assembly: intelligent robots efficiently assemble numerous building blocks that are optimized for being handled by robotic grippers.

3.1. The SL-Block System

The SL-Block, consisting of one S-shaped and one L-shaped tetracube joined along their sides, is an Octocube that includes eight cubic units. Two SL blocks are coupled to form a conjugate pair (Figure 2a). Unlike the 3D geometries designed in wood joints, the SL-Block’s shape appears structureless, lacking obvious boundary conditions that suggest the ways they can be joined. To address this challenge, we devise a formal grammar as an abstract tool for describing and analyzing the combinatorial and generative process within the SL-Block system.

The process of combining two geometries requires well-defined parameters for how one geometry translates and rotates relative to the other. Through analyzing this relative translation and rotation in the case of the SL block, we identified six transformation rules, each symbolized by a letter (Figure 2b). Employing each transformation rule allows the conjugate pair to be oriented into a new position, engaging with the original pair to form a combination. We termed this engaging behavior as SL engagement. With the help of these definitions, we can specify the intricate process of combining SL blocks by following a sequence of letters to generate a collection of the transformed conjugate pairs. The collection is referred to as an SL strand and can be represented as a string (Figure 2c).

3.2. Combinatorial/Computational Design Methods

To enable the computational generation of SL strands, we represent the six transformation rules using 4 × 4 matrices (i.e., [Mh, Ma, Md, Ms, Mt, My]). This matrix representation (Figure 3a) is well suited for computation and allows for the easy combination of rotation and translation described in each SL engagement.

When translating a string input into SL strands, the first step is to convert a letter sequence into a sequence of matrices. For example, the string sequence “hadsty” corresponds to [Mh, Ma, Md, Ms, Mt, My], and “hhdd” corresponds to [Mh, Mh, Md, Md]. Given such a matrix sequence (S) and its length (n), we can denote them as Sn. The union of all resulting transformations is denoted as TR. In the iterative process (Figure 3b), our goal is to ensure that any subsequence [TRi, …, TRn], for i = 1…n, is obtained through recursive non-commutative multiplication of matrices (i.e., TRi = TRi-1 × Si-1). Notably, if i = 1, TR0 is assigned the 4 × 4 identity matrix. This algorithm, implemented in Rhino 7/Grasshopper (Figure 3c), allows users to manually input different string sequences to generate various SL strands as output, bridging the string formulation with geometric structures.

3.2.1. A Grammar-Based Computation for Generating Geometric Patterns of SL Strands

The string representation of SL strands provides a symbolic mechanism that translates intricate geometric combinatorics into sequences of characters. In this context, manipulating the syntactic pattern of the input string directly relates to geometric patterns. As such, the study of pattern generation in SL strands can be seen as string operations.

As a simple example shown in Figure 4, the word “aaaa” can generate a closed SL strand that forms a cube unit. Following the grammar rules defined in the L-system model, we can formulate the initial variable (axiom) as “aaXaa”, where the “aaaa” is the start symbol. The inserted alphabet X will be iteratively replaced by the designed production rule (X→hhX) during each interaction. As the number of generations increases, the output string quickly grows in size and complexity, enabling the derivation of the smallest closed SL strand into a larger structure. Since the production rule refers only to an individual symbol rather than its neighbors, the grammar model applied for rewriting the string of the SL strand is specified by context-free grammar.

Generator: To assist the design and construction of large and complex compositions of SL blocks, we present a set of SL grammars as the 3D form generator. This includes the initial string variables and polynomial expression (Figure 5a), enabling the derivation of SL strands through the recursive grammar. The grammar defines the domain of its initial variable. The derivation of a variable based on rewrite rules in the grammar would uncover uncertainty and downsize the domain of the variable. When the derivative process is fully uncovered, the initial variable can be evaluated to a specific type of SL strand.

Compiler: Furthermore, we introduce syntax-directed translation (SDT), which is a method of compiler implementation. The expressive power of context-free grammar is often restricted by the syntactic structure of the parse tree. SDT can add new actions to replace the rewrite options. In the derivative process of the initial variable, SL strands are similar in syntactic structure but different in geometric patterns (Figure 5b).

3.2.2. Constraint-Based Computation for Tilling 3D Voxels with SL Blocks

The grammatical approach mentioned above adopts a bottom–up control method, systematically generating various SL strands by changing the syntactical structure of strings. However, the primary focus is exploring rule-based aggregations of SL strands, limiting the applicability of handling global design purposes. To receive user-specified shapes as input and automatically convert them into interlocking assemblies of SL blocks, top–down control methods are essential.

By analyzing the relationships between SL strands, polyline paths and 3D polyominoes within the SL-Block hierarchy, we formalized three critical combinatorial optimization problems (Figure 6) that should be computationally solved. In the following parts, we will introduce each of these problems, providing a detailed understanding of the algorithmic approaches employed for problem-solving.

The Path Consistency Problem describes the translation of a given polyline path into a string sequence. The polyline is drawn within a 4 × 4 grid, with segments constrained to orientations of 0°, 45°, or 90° (Figure 7a). Upon dividing the grid into 2 × 2 units, the polyline is discretized into a lattice path (Figure 7b–d). It is crucial to note that this constructive principle is specified to SL engagements like “h”, “a”, “s” and “t”. Decoding polylines into strings involving “d” and “y” requires a further subdivision of the 2 × 2 grid into 2 × 1, meeting the directional properties embedded in “d” and “y”. The created lattice path allows a directed graph to be built, where nodes represent point indexes and edges are vectors between them. Based on this graph, we employ a backtracking search algorithm that can decode the vertex sequence into a valid letter sequence. The algorithm operates as follows: (i) At each step, a letter is selected, and its value is assigned the cumulative transformation. (ii) For each transformation value, the algorithm inverses the value and checks consistency with the resulting transformed vector and the first vector in the given path. (iii) If consistency is confirmed, a recursive call is made for the next step. (iv) If there is any inconsistency, the algorithm backtracks. This iterative search process continues until a valid string (Figure 7e) is found for generating the matched SL strand (Figure 8).

The Hamilton cycle problem describes the search for closed lattice paths in given 3D polyominoes. The graph built from the 3D polyomino is a square grid graph whose vertices are interconnected by edges and arranged in a regular pattern. One common method used for identifying a Hamiltonian cycle in this type of graph is the backtracking algorithm. This algorithm starts from a selected node and always tries to extend the path by gradually adding the visited node and one of its unvisited neighbors. If the extended path does not allow the Hamiltonian cycle, the algorithm returns by removing the last added node and restarting the search process with another unvisited neighbor. The search process only concludes when a Hamilton cycle has been found or when it is clear that no Hamilton cycle exists in the 3D polyomino. Although the implementation of the backtracking algorithm makes it possible to create a closed path, major challenges arise with a larger search space. The backtracking algorithm with a brute-force search strategy verifies all possible neighbors of the vertex.

Using an algorithm that gradually selects the subsequent state that promises the best result at the time of selection (greedy algorithm) (Figure 9), we search for the Hamilton circle according to the following steps: (1) Subdivision of the grid cells: Each cell of the 3D polyomino is subdivided into four smaller units. (2) Creating a square grid graph: Creating an undirected graph from the center points of the smaller squares. The vertex denotes the point index, and the edge connects the vertex to its closest neighbors. (3) Prune edges using minimum-cost spanning tree (MST): Use the generated path to prune the edges of the graph that intersect with the path. (4) Weight the remaining edges: Based on the distance between their midpoints and the path, each remaining edge is assigned a normalized distance value as a weight. (5) Perform a greedy search of the Hamiltonian circle: Define the starting point and integration of backtracking into the search process by gradually extending the path by the unvisited neighbor whose edge has a higher weight value. (6) Return a list of ordered point indices: This list can be used to rearrange the center points of the original small squares to construct a closed polyline path within the 3D polyomino.

The resulting polyline is decoded into a string sequence using the algorithm developed for the path consistency problem, which can generate the corresponding SL strand. The algorithm we developed for finding a Hamiltonian can efficiently compute the optimal closed path from a large number of 3D polyominoes, even when the number exceeds thousands.

The 3D polyomino tiling problem investigates how to decompose a given shape into a group of interlocked 3D polyominoes (Tetris-like Polycubes). In the computational process, we can consider SL strands as individual solid polyominoes, breaking away from the combinatorial logic of SL blocks. To characterize their connection mechanisms, we summarize three types (Figure 10): (1) Joinery connections: akin to wood joinery, especially types like English dovetail, Halved joints, mortise and tenon. (2) Nesting connection: A unique interlocking mechanism where a set of SL strands is arranged in such a way that each strand is encompassed by another one, all the way to the outer strand. This mechanism is exemplified in Russian matryoshka dolls. (3) Borromean connection: This creates a network of interlinking loops formed by SL strands where the space of one loop strand is compensated by the other loops. These three connection types can offer a fundamental strategy for designing high-order interlocking structures with SL strands.

3.3. Architectural Speculation

Considering SL blocks as generic building blocks challenges conventional construction notions, rephrasing the resolution and syntax of modular architecture. Building modules previously considered as a whole can now be discretized into a collection of functional building blocks. These blocks can be easily assembled, disassembled, and reassembled into various building configurations. Instead of the originally planned enlargement of the SL block as a component, we introduced element hierarchies: SL-Blocks > SL-strands > building components (beam, column, wall, ceiling) developed (Figure 11a). This means that larger components are not created through larger-scale SL blocks but through their hierarchical aggregation on the original scale. This increases the resolution of a design and also the flexibility of reusability (Figure 11b,c). Compared with automating the assembly of buildings with various types of components, the additional work involved in assembling repetitive elements could be significantly reduced. A cuboid SL strand can represent a support or a beam. The hierarchical design principle enables the assembly of modular elements in different scales and also in combination with conventional building components. Several SL strands can be joined together to form large structures. At this hierarchy level, the complex SL block interlocking no longer plays a constructive role. For example, a cuboid made of 500 SL blocks can be regarded as a monolithic part that is conventionally connected to other parts.

3.4. Robotic Pre-Assembly Methods

3.4.1. High-Level Controller: Automated Planning for Robotic Assembly

To plan the robotic assembly of SL blocks, the assembly sequence must first be developed in such a way that all blocks can be placed without the already produced assembly being an obstacle for the next block. Collisions during assembly must be avoided. In addition to the sequence and direction of assembly, suitable grasping points for the robot need to be found to prevent collisions and enable a stable joining of the blocks. The automation of (dis-)assembly sequences is inspired by recursive algorithms developed for interlocking 3D puzzles [51]. In the digital model, we know the position of each block and use this information to find possible directions of movement for assembly. Here, the SL blocks are represented as graphs [52] that help to identify SL blocks that restrict the movement of their neighbors. Temporary fixation blocks were used during assembly to avoid undesired movements during their placement. Using a recursive algorithm, the sequences and directions for both assembly and disassembly can be determined.

We test our algorithms for robotic process planning through the assembly of various SL strands (Figure 12), which are designed to form walls, columns, and hollow cross-sections. Due to the limitations of pre-programmed robots, we operate our experiments with 3D-printed SL-blocks. These blocks have rounded edges that help with robotic self-calibration during grasping and placement.

3.4.2. High-Level Controller: Autonomous Planning for Robotic Assembly

To take automation a step further, we explored an autonomous strategy for planning the robotic assembly of SL-Blocks. In contrast to the above-described automated approach, which operates with pre-programmed instructions, the robot here has no prior information or commands about which blocks to use, their final poses, or the execution sequences. The only available information is the building blocks and abstract 3D target shapes. Therefore, the robot agent must autonomously discover the possible block combinations, find feasible action sequences, and put them into assembly practices.

To approach this challenge, we formulate the problem of combining building blocks into a target shape as a Markov Decision Process (MDP). The desired shape to be built is voxelized into grid cells. The state space includes four sets: (i) the set of placed blocks; (ii) the set of unplaced blocks; (iii) the set of target grid cells that should be occupied to form the desired shape; (iv) the set of non-target grid cells should remain unoccupied to ensure accuracy. For block placement, we use a discrete, time-varying action space. Each unplaced block can be positioned in any available grid cell, with four rotation options: 0°, ±90°, or 180° around the z-axis. A termination action is included to stop the assembly process. After each action, the states of placed/unplaced blocks and target/non-target grid cells are updated, and a reward is assigned. The reward is positive if target grid cells are occupied and negative if non-target grid cells are filled. A successful placement action requires the block to be placed without collision and in a stable configuration. An invalid action results in the episode being terminated with a high negative reward. Otherwise, the episode terminates under any of the following conditions: (i) the agent chooses the termination action, (ii) no more available blocks, or (iii) the completion of training.

Developing such an MDP framework requires algorithmic methods that can (i) handle the combinatorial search space of selecting the block type and optimizing its placing poses to resemble the desired geometric shape; (ii) consider the robotic constraints, collision avoidance and structural stability for sequencing robotic tasks and motions. To address the upper level of complexities, we developed a model-based mixed-integer linear program (MILP) [53] and a learning algorithm that combines the graph neural network (GNN) [54] and Q-learning [55] (i.e., model-free reinforcement learning).

In the MILP problem, we mainly focus on the combinatorial optimization of the block’s position and orientation to configure the desired geometric shape. We developed a MILP solver that can generate the optimal values for the decision variables, revealing the final poses for each type of placed block. In Q-learning, the use of GNN provides the required representational flexibility for encoding the environment’s state as a graph, where nodes represent all blocks and grid cells, and embed features (e.g., spatial coordinates, constraints, etc.) that inform the learning process. The graph creation is followed by three rounds of message passing using an attention mechanism, in which we sequentially build an encoded graph. The encodings are the basis for computing the Q-value for predicting the action’s quality. Through the GNN-Q learning, the robot can decide executable action sequences from the set of actions proposed by the MILP solver (Figure 13). It is important to note that the long-term horizon of the task is taken into account, as we can only judge at the end whether all actions optimally reproduce the desired target shape. For more details about the methods, please see [56,57,58].

There are three important results achieved here: (1) We develop a MILP solver that can generate the combinatorial rules from the available blocks and arrange them into the desired shape. (2) We develop an integrated long-term planning algorithm that learns through exploration, which in combination with a MILP solver leads to generalizable capabilities. Our method considers the robot in the process and integrates action planning with motion generation to learn control processes that ensure the kinematic feasibility and stability of the constructed structure. (3) Finally, we develop a novel benchmark environment for autonomous robotic assembly that is modular for testing with and without the robot, for arbitrary target designs, and with adjustable types and numbers of building blocks. Our empirical results from a series of representative experiments demonstrate the generalization capability of the proposed algorithmic solution and provide interesting insights into the combination of integrated learning and planning for long-term robotic manipulations that can contribute to a range of robotic applications. Tasks can be completed without the robot having prior knowledge of detailed instructions. The system captures the spatial relationships between the target shape and unplaced blocks, enabling autonomous decision-making. Our hierarchical controller performs high-level learning of task plan and goal specifications (construction) together with low-level action sequences. This allows tasks with a long-time horizon to be executed, where many actions (inserting small elements) contribute to the completion of a large task (overall assembly). The system also supports the disassembly of the built structure. Once the blocks have been placed, their exact position and the action sequence can be determined, allowing the robot to reverse the steps if necessary.

3.4.3. Low-Level Controller: Learning Sensor-Based Perception and Manipulation Skill for Robotic Insertion

To compliantly insert a SL-block into another, a learning-based low-level controller needs to be developed that can handle the complexity of physical contact and frictional challenges at smaller scales. This controller is reactive and responsible for sending commands directly to the robot arm and end effector to receive feedback. Based on this setup, we explored unsupervised representation learning of tactile sensory data for high-precision insertion tasks. The experimental setup involves a UR 10 robot with a Robotis gripper and DIGIT tactile sensors [59], combined with OptiTrack cameras and AprilTag tracking. In the lab environment for robotic learning experiments, a real robotic environment needs to be connected to multiple hardware components, each requiring individual drivers and setups. Therefore, we established this system in an Assembly Gym. The Assembly Gym Framework is a software layer that represents contact-rich insertion tasks as a reinforcement learning problem. This allows us to apply existing RL algorithms while enabling the use of any simulator or real robot as the “environment” in which the RL algorithm operates. The architecture and key design concepts are presented below to enable the adoption of this setup:

(1) Hardware Components: The physical components consist of the robot manipulator, the tactile sensor (https://digit.ml/ (accessed on 23 July 2024)), and the tracking system. The used robot manipulator is a 6DoF UR 10 robot arm (Universal Robots, Odense, Denmark) with a Robotis RH-P12-RN-A gripper (Robotis, Seoul, South Korea on the end effector. The Robotis RH-P12-RN-A is used because it provides a low-level control API that allows the gripper to be activated by current, which is necessary when attaching tactile sensors to it. The gripping force must be set accordingly so that the silicone gripping surface of the tactile sensor is not damaged. The DIGIT design of the tactile sensor was chosen because it is open source and can be produced in-house at a low cost. As it is produced in-house, it is possible to make improvements and adapt it to specific needs. Furthermore, assembling multiple objects into a structure usually requires some form of object localization. For this purpose, the setup offers two tracking systems: an RGB camera in combination with AprilTag fiducial markers, and OptiTrack, a precise motion detection system based on infrared cameras and reflector markers.

(2) Software Architecture: For technical reasons, the implementation was split into two code bases, which run on different machines. One code base contains all the Python code that implements the user interfaces, while the other contains all the back-end and driver-level code within the ROS framework. Our software design follows the composition pattern (Figure 14). The environment is instantiated as a hierarchy of components, with the structure of components visible on the highest level. We decompose our BaseEnv into the following abstract components (Figure 15):

• Controller, sensors, reward, and task: these components are called by the BaseEnv and receive the environment in which they operate. The controller receives an action, and the environment executes this action by calling the environment. The sensors and the reward receive the task and extract and return observations and a reward. The task is the most complex component as it defines the task-specific behavior, such as ending the current episode or maintaining a state needed to compute the reward function, by implementing a step and reset function.
• Serializable Global Top-Level Configuration: All important configuration parameters are defined at the highest level in the user code and then passed through the component hierarchy. This centralized configuration is crucial in the research area, where the code is part of the experiment and parameters often need adjustments to achieve results. We want to ensure that the user has an overview of all parameters that influence the behavior of the environment in one place. By preventing unintentional misconfigurations, the process of parameter customization can be accelerated. During experiments, logging these parameters alongside experiment results ensures context and reproducibility. Therefore, our global top-level configuration is serializable.
• Real and Simulation Environment: Designed as an abstract component, the environment can have multiple implementations that are used interchangeably. In this case, we implement both a Real Environment based on the real physical lab setup and a Simulation Environment based on PyBullet 3.21. Due to the challenges associated with simulating the DIGIT sensors, the physical setup is used first. While the Simulation Environment interacts with the PyBullet physics engine, the Real Environment connects to the drivers of the various hardware components via ROS.

Insertion Task Design and Reinforcement Learning: We consider a simplified setup for the robotic insertion task of SL blocks. In this setup, an already assembled structure is fixed on the table and a new SL block is inserted into the structure by the robot. Thus, the task is represented as moving the grasped SL block from its current position to the target position. The reward function is designed to minimize the distance of the SL block from its target position.

• Reward: The reward r is the negative distance d between the position of the moving block p and the target position p0. Since everything is transformed into the frame of the target position, the target is the origin p0 = (t_0,r_0). The distance between the positions is a weighted sum of the translation and rotation distance d(p,p0) = αd_transl(p,p0) + (1 − α)d_rot(p,p0 ), which combines two values with different units. Since translation distances are in the range of centimeters and rotations are in radians, we set α= 0.9.
• Action space: Constraining and exploring the action space is facilitated by Cartesian space control, as it is easier to visualize and understand than joint control. Originally, the aim was to achieve force or speed control in Cartesian space, but real-time control performance could not be achieved with the existing setup, so position control had to be used. Intuitively, force control in Cartesian space or torque control appears to be the most suitable control mode for the task. The torques and forces in Cartesian space exerted by the manipulator correlate directly with the tactile feedback.
• Observation space: The observation space is multimodal, as it consists of sensor images and positions of the gripper and the SL block. All positions coming from the environment are seven-dimensional vectors px, py, pz, ωx, ωy, ωz, ωw. The images from the DIGIT sensor are three-channel RGBs with a resolution of (w: 240, h: 320).

Reset: A fully autonomous reset of the environment is desired when a large dataset is collected. The reset procedure includes picking up the SL block, either by extracting it from the structure if the last episode was successful or by retrieving it from the table. As there is no re-gripping implementation, the human monitoring of the process is notified if the SL block rolls into an inaccessible position. With most repeated trajectories resulting in a successful insertion, human intervention is minimized. The transition function t: (o_n,a_n) →o_(n+1) is approximated by a neural network based on pairs of data (o_n,a_n,o_(n+1)) consisting of the nth observation o_n, the action on and the observation of the next time point o_(n+1) sampled from the system. Since feeding a complete image into a fully connected network is not feasible, the images must be compressed to a lower dimensionality by an encoder network, for which we used a convolutional neural network (CNN).

Data Collection from Human Demonstrations: The initial data for training an insertion strategy were collected from human demonstrations. AprilTags are used to track the SL blocks. Object tracking and detection is an active area of research with many different approaches including robotic joint measurements and computer vision-based object tracking [60,61].

We use AprilTag 2 as a visual reference system. The tags can be created with an ordinary printer and the AprilTag recognition software calculates the exact 3D position, orientation, and identity of the tags relative to the camera: The positions of multiple tags are averaged to improve the position estimation of the SL block (Figure 16). To learn the transition function of the system, a large dataset needs to be collected for offline learning. These data should cover as much of the relevant state space as possible. In our case, randomly activating the robotic system would not capture states during the critical joining phase when the block slides into its installation position. As such, we demonstrate and record hand motions for each connection type of joining blocks (Figure 17). The data are collected as a sequence of SL-block position measurements, which are then traced by the robot. To increase the variance of the data without having to record additional demonstrations, the recordings are noisy and are also interpolated to generate new trajectories. During execution, the robot adjusts its policy based on tactile feedback. As a result, each attempt at the task introduces new conditions, contributing to the dataset for training and self-improvement. This iterative process enables the robot to adapt and refine its performance over time. The real-system control for robotic insertion of SL blocks is implemented with a Cartesian velocity controller. Using a learned model of environmental dynamics, the system predicts and plans trajectories for execution. These plans are continually updated after each action is executed, forming a model-predictive control loop. The developed low-level controller correlates tactile readings with block positions and robot commands, enabling the Cross-Entropy Method planner to generate optimal action sequences that maximize rewards, achieving successful block insertion (Figure 18).

4. Results—Demonstrator Design and Construction

The demonstrator is composed of four diagonals that form two V-shaped parts and originate from a central solid base. Viewed from above, the diagonals form four equally sized quadrants. A truss-like girder and a tension chord intersect and connect two opposite upper ends of the diagonals. SL blocks are aggregated to form SL strands, which in turn are joined to the overall structure using the three different connection principles for larger structures (i.e., joinery connection, nesting connection, and Borromean connection). The computational design of the demonstrator was initially formed by voxels in a spatial grid, the resolution of which was increased depending on structural requirements. Once the individual parts were represented as discrete cubes, they were transformed into SL strands. Using the Hamiltonian cycle-finding algorithm, we can form a closed polyline by connecting the geometric centers of the voxels. Each polyline is then decoded into a string sequence to generate the corresponding SL strand [52].

The individual parts (truss girder, diagonals, support, tension chord) were automatically assembled with industrial robots. The construction of these building parts was carried out in three phases (Figure 19). The assembly-related plan was mapped out through user tests and experiments in our developed VR framework (Figure 20). The construction exposes the SL-Block assemblies to compression, tension, and bending to test the performance of the interlocking system. A solid plinth absorbs compression forces; V-shaped columns bend and are tied back by a modular truss. Truss girders are post-tensioned using different approaches. The local differentiation of SL-Block materials reflects the different local forces: Regions exposed to compression are made from concrete. Spanning regions are made from lightweight SL-Blocks.

5. Discussion

The topological feature (e.g., symmetry, connectivity, complementary) of geometric constituents in the SL-Block establishes a set of six connection rules. Through these six connections, the serial repetition of a single small-scale SL-block can be hierarchically organized into larger building parts or components, resulting in a complex functional whole and diversity. The self-interlocking of SL-blocks allows for reversible joining and can significantly minimize the need for additional fasteners (e.g., screws), thereby reducing assembly heterogeneity. This element-based method simplifies the design and assembly process, enabling the computational and mathematical description of a modular building to become feasible. Theoretically speaking, in architecture, our work on the SL-block system will contribute to research on discrete and digital [14,16], combinatorial design [9,15], and topological interlocking assembly [20]. Additionally, our findings can extend to broader fields, including programmable matter, “digital materials”, self-assembling materials [12,13,22,62], and reconfigurable interlocking assemblies in computer graphics [24], and so on.

Notably, when the complexity of SL-block combinations crosses a certain threshold, that complexity can be isolated by defining a separate abstraction that has a simple interface. The abstraction hides the complexity of the element, while the interface indicates how the elements interact within the larger system [63]. Our study on the SL-block system developed a symbolic representation for six connection rules and grammar-based string operation to hide the descriptive complexity of the SL-block combinatorics, facilitating the computational generation of SL-strands (Section 3.1 and Section 3.2.1). Moreover, we abstracted the SL-block assemblies and SL-strand geometries into the polyline paths and solid polyominoes, which are prerequisites for developing constraint-based computational tools that can automatically convert the user-specified 3D polyominoes into the interlocking assemblies of SL-blocks (Section 3.2.2). In this way, design complexity becomes manageable through the polyomino and polyline-path representation of SL-strands, along with three new connection types that show how SL-blocks interact to form the large-scale assemblies (Figure 10). As a result, we believe that the abstraction [64] is a useful means for the computational analysis and design of hierarchical interlocking assemblies.

The high-level controller developed for the robotic assembly system presents the assembly planning algorithms that coordinate the geometric-based assembly sequence [65] of SL-blocks with robotic parameters, ensuring collision-free robotic paths and motions throughout the global assembly process. To further enhance assembly efficiency, additional study is needed on assembly line balancing [66] to group assembly operations of SL-blocks. Beyond the assembly from disassembly principle [67], this study highlights the potential of using model-free reinforcement learning to train a robotic agent in autonomously discovering the design and assembly plan of target structures with the only available information—the building blocks and an abstract envelope shape. Our learning-based algorithm was successfully carried out in simulations and a physical test environment with 22 polycubes. Further research is required to scale up the limited number and types of blocks, thereby enhancing the generalization [68,69] of this autonomous robotic approach to design and plan interlocking assembly.

Within the low-level robotic controller, we develop the deep reinforcement learning framework, which is specifically tailored to train robots in acquiring precise insertion skills to handle the complexity of high-frictional rigid contacts. Central to this achievement is the accurate estimation of contact forces, pressure and torques, which inherently necessitates advanced sensor techniques. The integration of the visual–tactile sensor with the robot complements current research on learning-based robotic insertion. While traditional approaches rely on feedback from force–torque sensors [34] and camera-based object tracking and recognition [38], our incorporation of tactile sensors enhances the robot’s sensory capabilities. During the insertion process of SL-blocks, the system learned, in a completely unattended mode, the representation of tactile sensor images from which contact forces, pressure, and torques can be derived. This representation is the first step towards exploring generalizable tactile representations for the widespread application of visual tactile sensing in architecture and construction. Moreover, as part of the project, we trained a classifier for surface textures for the DIGIT visual–tactile sensor to recognize the material properties (e.g., shape, hardness) of wood or concrete SL-Blocks. This process entailed creating a dataset of objects with different surface properties and training a model on it. Consequently, the DIGIT sensor achieved an impressive 90% accuracy in texture recognition, while the recognition rate of textures by human test participants was just 67%. This outcome underscores the potential of visual–tactile sensors to extend beyond insertion processes, empowering robots to autonomously manage surface quality and sort various building materials. We have to point out that the developed robotic insertion controller still faces challenges arising from cumulative tolerances, especially when assembling a large number of SL-blocks. Further research is required to scale tactile grippers for lifting larger loads. Here, the Shore hardness and material thickness of the silicone layers must be adapted. The visual interpretation of the deformations can be scaled and generalized.

In this project, the final assembly of SL-strands into a cohesive structure was carried out through human collaboration. To advance on-site construction automation, further studies on robotic collaboration (e.g., collective robotic assembly) [42,43] or human–robot collaboration via large language models (LLMs) [70] are needed. Additionally, exploring parallel construction strategies [71] is necessary to enhance construction efficiency.

Further questions for future research arose in the course of the project. Which manufacturing technologies ensure the necessary precision of self-calibrating SL blocks at a reasonable cost? How can the geometrical shape of SL blocks be further developed so that the locked degrees of freedom are adapted to the respective installation position and the associated load-bearing cases? How can the necessary balance between efficient (dis-)assembly and high-contact gearing be optimized? How can the joining and combinatorial principle of SL blocks be topologically reasoned to facilitate their transfer into industrial modular practices?

6. Conclusions and Outlook

This article proposes an integrated approach to the combinatorial design and assembly automation of modular buildings. Instead of considering buildings as permanent structures, we focus on designing modular building elements that can be reused in a series of building types, adapting to the changing needs of our society. The conclusions of our work are as follows:

• Modular Element and Architectural Speculation: SL blocks are suitable for mass customization in architecture. Their combinatorial versatility and complex interlocking logic allow for design diversity and building configurations that require almost no temporary support structures. These properties result from the geometric feature of the blocks, which makes their production costly. Industrial mass production of the SL blocks is possible, as there is no parametric differentiation of the parts. This means that a small number of molds/forms can be reused or other manufacturing processes can be standardized and optimized for one geometry. However, SL-Block constructions are intended to achieve the different architectural and structural requirements of a building using blocks that have different properties with different materials. The complex geometry would therefore have to be produced using a wide variety of processes. High precision must be ensured in all manufacturing processes so that tolerances do not accumulate and the force transmission between the blocks functions smoothly. Future research will focus on developing serial building blocks based on the studied principles from the SL-Block system. These new building blocks should address the limitations of the SL-Blocks and be optimized for industrial constructions. Parallel to working on the project, we were able to observe how ideas regarding dry and reversible joining of “micro-modular” building blocks are emerging not only in research but also in practice. Reversible clinker brickwork and the use of wood offcuts to build wooden building blocks are new systems for which robot-assisted assembly could become relevant in the future.
• Design Computation and Robotic Assembly Planning: Hierarchical analysis and graph-based computation are important tools for simplifying the combinatorial design. Representing the geometric and topological methodology in graphs is crucial work for the automated and autonomous robotic assembly of building elements. Representing and analyzing the geometric degrees of freedom during assembly is important preparatory work for further developing the interlocking logic of the blocks, making future projects more practical.
• Learning-based Tactile Sensor Technology and Robotic Insertion Control: We have demonstrated that tactile sensing can be used to learn insertion strategies, reliably identifying the physical properties of different materials. However, the entire robotic learning process requires large amounts of trial-and-error data, which must be obtained from physical experiments. A major challenge in this process is the difficulty of simulating physical parameters, such as contact between the gel pad and SL blocks, which requires a highly precise digital-twin model. Further research in deep generative modeling is needed to create more detailed and realistic models for object tracking. The use of such simulations for state estimation of multiple building blocks will significantly accelerate the robotic assembly process. In the future, more modular and hierarchical learning methods are needed to manage different blocks and assembly types. The vision–language–action (VLA) model may offer a promising direction to achieve task generalization by focusing on the object-level inference instead of performing direct control via image-to-torque translation, as with the DIGIT sensor. Similarly, there is a need to incorporate tactile perception into such a foundation model. The research on autonomous robots in pre-assembly lays the groundwork by enabling robots to perceive their environments and make independent decisions. This capability forms the basis for achieving the collaboration ‘at eye level’ between humans and robots.