Development of Architectural Object Automatic Classification Technology for Point Cloud-Based Remodeling of Aging Buildings

Abstract

In this study, we address the challenge of efficiently handling the maintenance and remodeling of buildings constructed post-1960s, lacking architectural drawings. The conventional approach involves manual measurements and data recording, followed by digital drawing creation. However, we leverage Fourth Industrial Revolution technologies to develop a deep learning-based automatic object classification system using point cloud data. We employ the FCAF3D network with multiscale cells, optimizing its configuration for classifying building components such as walls, floors, roofs, and other objects. While classifying walls, floors, and roofs using bounding boxes led to some boundary-related errors, the model performed well for objects with distinct shapes. Our approach emphasizes efficiency in the remodeling process rather than precise numerical calculations, reducing labor and improving architectural planning quality. While our dataset labeling strategy involved bounding boxes with limitations in numerical precision, future research could explore polygon-based labeling, minimizing loss of space and potentially yielding more meaningful results in classification. In summary, our technology aligns with the initial research objectives, and further investigations could enhance the methodology for even more accurate building object classification.

1. Introduction

Through the Industrial Revolution, mankind became capable of mass and automatic production through the rise of technological power and the development of machinery. Likewise, in the world of architecture, countless buildings are built. However, after World War I and World War II, many buildings were destroyed and lost, and people began to build residential spaces haphazardly. In particular, in the architectural industry, the invention of concrete led to the construction of structurally strong buildings, and such buildings have been produced in countless numbers since the 1950s. Now, over 60 years later, these buildings require remodeling, which is becoming more common through urban regeneration initiatives [1].

The conventional method for creating drawings before demolition relies heavily on manual labor. Workers use tools like rulers and laser measuring devices to measure dimensions and manually sketch architectural elements such as walls and columns. The data collected through this process is then used to create digital drawings, which, in turn, facilitate structural assessments and architectural planning to identify the components that need to be removed, contributing to waste reduction.

In the era of the Fourth Industrial Revolution, various smart technologies have emerged, finding applications across industries, including architecture [2].

These technologies encompass automated recognition and generation of drawings from data, as well as the use of deep learning for classifying building objects.

Smart devices like 3D scanners and drones, which employ laser-based point cloud data, have expanded the scope of work in the industry, allowing for tasks previously impossible with traditional 2D photography.

These technologies are particularly useful in indoor architecture for classifying and automatically arranging objects. The Fourth Industrial Revolution has the potential to replace labor-intensive, repetitive tasks, such as the manual measurement process, with the efficiency of machine and digital elements. This shift can free up time for more creative endeavors. Consequently, this research aims to develop a deep learning-based object classification technology using point cloud data to replace labor-intensive methods when remodeling aging buildings [2].

2. Background and Related Research

As mentioned, during the architectural measurement process, the step of creating architectural drawings separates the objects to be demolished from those to be preserved. To do this, a structural assessment phase is typically required. However, if we can automatically classify architectural objects before this step, it becomes easier to calculate the subsequent architectural waste and identify structural elements to be preserved. To achieve this, we aim to develop deep learning-based object classification technology, which requires the prior collection of training data.

In this research, we intend to classify architectural objects such as walls, columns, floors, and roofs, with an additional category for miscellaneous objects not included in actual buildings. To accomplish this, we plan to utilize point cloud public data from AI-Hub, provided by the Korea National Institute for Artificial Intelligence (AI-Hub), and real 3D scanning data of aging buildings. The methodology of this study is illustrated in Figure 1, as follows.

First, through an analysis of the architectural remodeling process, this study aims to determine at which stages our developed technology can be applied and identify the necessary elements. Subsequently, a preliminary analysis of prior research on point clouds is conducted. Then, the deep learning network to be used for training and classification in this study is analyzed and selected. Further modifications are made to adapt the chosen model to the specific needs of this research.

Following this, the collection of point cloud data is carried out using the aforementioned public data and actual measurement data. This data often contains unnecessary information when used for training, as it includes numerous data points, which significantly increase data volume compared to regular image data. Therefore, a data refinement process is applied to the point cloud data, followed by the training phase.

Finally, the trained model is used to verify the classification of architectural objects into their respective classes. This process is aimed at developing detailed construction techniques that can efficiently utilize the classified architectural objects in the later stages of the building remodeling process.

3. Analysis of Aging Building Remodeling and Object Recognition Technology

3.1. Guidelines for Aging Building Remodeling and Analysis of Related Research

Remodeling is defined as “the act of suppressing the aging of a building or partially expanding it for function enhancement, among other things”. In the context of extending the lifespan of a building during its lifecycle, remodeling is a crucial category in architecture, offering an efficient means to establish a sustainable architectural environment in the 21st century [3]. Currently, terms like remodeling and renovation are used interchangeably, but this text predominantly employs the term “remodeling” in a broad sense.

The remodeling process can be divided into planning, design, and construction phases. The planning phase is particularly important in the execution of remodeling projects, as it requires a more detailed analysis and process than new building construction plans to maintain and enhance various building performances [4].

Recent research directions in the field of aging building remodeling can be broadly categorized into three areas. First, there is research aimed at improving energy efficiency, with various studies exploring ways to reduce energy consumption in aging buildings and investigating energy-saving potential. The second area of research involves the study of recycled materials and building waste, focusing on sustainable remodeling materials and waste management practices, aiming to reduce environmental impact. Lastly, there is research on the introduction of renewable energy, such as solar and wind power systems, to enhance the sustainability of aging buildings, as seen in Figure 2.

Notably, there is a vibrant body of research in the planning phase of remodeling aimed at enhancing energy performance. These studies typically conduct phased analyses based on the building’s lifecycle and prioritize analyses depending on the extent of building reuse during remodeling.

Interest in aging building remodeling is also growing in South Korea, and guidelines have been established accordingly. The Ministry of Land, Infrastructure, and Transport has published the “Guidelines for Aging Building Remodeling”, which focus on safety, energy efficiency, and cultural heritage preservation. Additionally, the Korea Conformity Laboratories (KCL) provides technical guidelines for aging building remodeling, emphasizing energy efficiency and safety.

Internationally, several guidelines have been introduced. Notably, the International Union of Architects (UIA) offers international guidelines related to aging building remodeling, emphasizing cultural heritage preservation and sustainability. The U.S. Green Building Council (USGBC) provides LEED certification for improving the sustainability of existing buildings.

These guidelines are formulated to enhance building functionality, safety, and sustainability. To achieve this, architectural drawings and measurement data play a crucial role in the remodeling process.

Currently, approximately 31.3% of buildings in South Korea do not possess architectural drawings. This percentage is derived by dividing the number of buildings with registered drawings in the Road Address Building Data held by the National Geographic Information Institute by the total number of buildings.

During remodeling, architectural drawings are essential to understand the structure, layout, and facility locations of the existing building. In the absence of drawings, data collection must be done through measurements, particularly at the site, to precisely comprehend the current state of the building.

The measurement process for creating architectural drawings typically begins with preparing laser distance measurement devices, cameras, scanners, and safety equipment. Both the interior and exterior of the building are measured, capturing vital architectural elements like size, structure, walls, windows, doors, and stairs. Detailed measurements are taken for ceiling height, door width, window size, and the location of electrical and plumbing systems. All data, including photographs, is documented, enabling the generation of digital data. This data is then analyzed and processed in real-time to ensure accurate results.

Measurement techniques can be categorized into three methods: manual measurements, laser measurements, and photographic measurements. These methods vary in the tools they use, but they all involve extracting drawings based on the measured dimensions through on-site sketching and documentation [6]. In recent years, the development and advancement of Fourth Industrial Revolution technologies have led to increased efforts to utilize tools like 3D laser scanners, drones, and image-processing devices [7]. Figure 3 provides an overview of a proposed 3D object measurement tool equipped with a topographic LIDAR view.

These methods enable accurate and rapid measurement of a building’s structure and shape while capturing the external and internal aspects of the building at high resolutions. Therefore, in this study, we aim to develop technology that utilizes such devices and techniques to perform the measurement process in aging building remodeling more efficiently.

3.2. Analysis of Deep Learning-Based Point Cloud Object Recognition Technology

Point clouds and triangle meshes serve as robust representations, preserving intricate surface details essential for conveying comprehensive geometric information related to the structure of buildings and building components, as well as their interconnections [8]. The advent of deep learning has notably enhanced object recognition within this context. Deep learning leverages its capability to acquire prior knowledge of object features by performing extensive feature calculations on substantial datasets in advance [9].

The method for searching for objects within point cloud data is not significantly different from searching for objects in traditional images using networks [10,11]. In conventional images, objects are labeled with Bounding Boxes in x and y coordinates, and networks, primarily based on Fully Convolutional Neural Networks (FCNN), are used to locate objects in new images. The method for searching for objects within point cloud data is fundamentally similar [12].

Recent state-of-the-art methods for object recognition include point-cloud object recognition based on histograms of dual deviation angle features (HDDAF) [13] to detect unknown objects in point clouds by matching histograms of unknown objects with a model library. Shi et al. [14]. Developed an effective and efficient 3D object detector using a voxel-point geometry abstraction scheme for acceleration and the generation of proposals.

The previously mentioned network, FCNN, is one of the key networks that has rapidly advanced the development of deep learning technology. It is effectively used for object segmentation in high-resolution images. FCNN is an extension of traditional convolutional neural networks designed for image segmentation tasks [15]. This network has significant applications in semantic segmentation, object detection, which accurately predicts object positions and boundaries, and natural language processing, spanning fields such as roads, autonomous driving, medical image analysis, and more.

Semantic segmentation in point clouds is also quite similar to semantic segmentation in 2D images [16]. In traditional 2D images, labeling is done pixel by pixel for training, while in 3D, points are labeled similarly to pixels for training. Figure 4 illustrates the structure of FCNN. The foundational structure of FCNN involves augmenting the VGG 16-layer convolution network with a multilayer deconvolution network. This addition aims to produce a precise segmentation map for a given input proposal. Utilizing the feature representation derived from the convolution network, employ a sequence of unspooling, deconvolution, and rectification operations to construct a dense pixel-wise class prediction map [17].

Among the deep learning models used for processing point clouds, PointNet is currently one of the most prominent models in terms of performance. The generic object recognition algorithm is: Achieve high quality and accuracy while maintaining high efficiency [18].

PointNet efficiently addresses the structural disorder inherent in point cloud data by realizing effective end-to-end feature learning. PointNet avoids data transformation and handles displacement invariance of input points [19]. Additionally, PointNet has been successful in tasks such as segmentation and object classification, outperforming previous methods such as PointNetX.

PointNet is highly effective in performing tasks such as 3D object classification, segmentation, and recognition by extending traditional 2D image processing techniques into the 3D space [20]. PointNet takes 3D data represented as points or point clouds as input. This model learns the features of each point and integrates them to recognize or segment objects. One notable feature of PointNet is its ability to generate consistent results regardless of the input order.

PointNet, based on the principles shown in Figure 5, was first introduced by Qi et al. in 2017. Subsequently, an improved version called PointNet++ was developed [21]. While the initial version focused on processing single-point clouds, PointNet++ was designed to handle high-resolution point clouds through a hierarchical structure, resulting in enhanced performance in capturing spatial hierarchical information. These models are effectively utilized in the classification of 3D objects and the recognition of 3D objects and boundaries [20].

The lack of digital representations impedes the efficient operation and renovation of existing buildings. Digital information in the form of geometric information can be extracted from the shape of buildings and a high degree of surface detail in triangular meshes and point clouds.

In this study, we developed a technology that can efficiently be utilized in the remodeling process for numerous buildings without architectural drawings. Specifically, we employed point cloud data obtained through 3D scanning to classify architectural objects using the FCAF3D model, making it applicable for utilization.

The bottom-up approach refers to recognizing individual points or small parts first and then combining these small parts to recognize the entire object. It involves extracting features for small parts of 3D points and clustering or grouping points with similar features. By combining these grouped points, it recognizes and segments the entire object. While this approach performs well in finding details, it is challenging to recognize the overall structure, and it has a high computational complexity.

On the other hand, the top-down approach involves first recognizing the overall structure or large objects and then decomposing them into detailed parts for recognition. In the case of FCA3D, it classifies objects first and then proceeds with instance segmentation through learning. This method is effective in quickly recognizing large objects, but it may miss detailed details.

This research aims at the classification and segmentation of point cloud datasets to increase automation in the accuracy of the classification of architectural objects in the context of the renovation of aging buildings. The scope of architectural objects in this research encompasses point cloud-built objects scanned for renovation.

4. Construction Object Automatic Classification Dataset based on Point Cloud Data Construction

4.1. Building Object Segmentation Dataset Construction for Architectural Object Classification

The following Figure 6 represents the process of this study.

The majority of the training data used in this study was sourced from the public dataset “Indoor Space 3D Comprehensive Data” available on AI-Hub, as mentioned earlier. This dataset encompasses indoor space data from various types of buildings, including regular housing (apartments, villas, etc.), as well as special facilities such as hospitals, libraries, factories, amusement parks, and more. The indoor space data in this dataset was divided into Bounding Boxes, and the labeling was provided in ‘JSON’ format. Among the numerous data sets available, an example set of 3D point cloud data and ‘JSON’ files is illustrated in Figure 7.

Furthermore, actual scanned data was obtained from the “Bokhyeon 1-dong Refugee Village Documentation Project” conducted in the Bokhyeon 1-dong area of Buk-gu, Daegu, South Korea. This data includes 3D scanning data of aging buildings (houses) located within the project area. An example of this data is shown in Figure 8.

Based on this point cloud data, labeling was carried out for the categories of walls, columns, floors, and roofs using bounding boxes, as mentioned earlier. Other elements were labeled as miscellaneous categories. The labeled data includes the smallest and largest x, y, and z values for the coordinates of each object. When using this labeled data for training, it is important to consider that multiple objects may overlap and be present within the same point cloud. This can lead to the possibility of mixed or overlapping objects when modeling these points. Therefore, a preprocessing step similar to conventional image segmentation is necessary to prevent such situations. In other words, it was necessary to relabel the existing AI-Hub “Indoor Space 3D Comprehensive Data”, and an example of the newly labeled data is shown in Figure 9.

The data listed in Figure 8 represents individual points that have been labeled, and these points are organized into an “array” for labeling.

4.2. Building an Object Classification Network Establishment

For training, we based our network on ‘FCAF3D’ (https://github.com/SamsungLabs/fcaf3d, accessed on 7 July 2021) [23]. While there are other networks like PointNet and SSPC-Net for classifying 3D objects, the ‘SubRGBD’ dataset is widely used as an evaluation metric for the performance of 3D object classification networks in the fields of computer vision and networks. The term ‘SubRGBD’ is an abbreviation for “Subtractive RGB-D”, signifying RGB-D data processing and analysis techniques. RGB-D data typically refers to data types that combine color (RGB) and depth (D) information. This type of data provides depth information in 3D space alongside 2D images [24]. It denotes the methods used to extract desired information or eliminate unnecessary information from a given RGB-D dataset.

The FCAF3D network has been exhibiting the best performance on the ‘SubRGBD’ dataset, and it offers reasonable training and model speed. It operates using units referred to as ‘voxels’, which are concepts that correspond to pixels in 2D images [12]. Voxels are used as virtual minimum units in 3D datasets, and the training allows for adjusting voxel sizes. In this study, we used a base unit size of 1 cm. Figure 10 illustrates the definition of voxels in a visualized form.

Furthermore, Figure 11 represents the fundamental network architecture of FCAF3D.

As mentioned above, FCAF3D’s model is not significantly different from the models for semantic segmentation of 2D images. It learns the data through 3D convolution and pooling processes. In this study, we stacked 4 blocks for training, but for more precise training, you can increase the number of blocks. Conversely, when precision is not required, you can reduce the number of blocks for training. Increasing the number of blocks may slow down the training process, but it offers the advantage of detecting smaller objects.

The overall framework of the FCAF3D is outlined, featuring three-dimensional and sparse convolutions and transposed convolutions. This specific design facilitates the processing of the input point cloud in a single forward pass.

5. Automatic Classification Model for Architectural Objects in Renovating Aging Buildings

5.1. Segmentation Dataset Training

The training was conducted by varying the model’s block count to 4 and 5 or changing the voxel size. When the baseline was set with 4 blocks and the voxel sizes were changed to 1 cm, 1.5 cm, and 2 cm, the training results are shown in Figure 12a–c, respectively.

The Epoch denotes the fundamental unit iteratively undertaken during the learning process. Notably, with the current representation of point cloud data characterized by points and possessing a three-dimensional structure, a pivotal adjustment has been introduced. Specifically, the unit of Voxel for learning has been redefined, and the epoch has been formally designated.

It was observed that the performance decreased when the voxel size was increased to 2 cm. Furthermore, it was found that the 1 cm voxel size produced the most accurate results. The reason behind this is that reducing the voxel size reduces the overlap of cells. However, when considering the entire cell, the overlapping area increases. As a result, the accuracy of training is expected to improve accordingly, and the results confirmed this observation.

Figure 12d,e above show the results of training with the same voxel size while varying the number of blocks.

Changing the number of blocks did not significantly impact the network’s performance, but it was observed that the training speed was faster when using 4 blocks. This is also because, when looking at the structure of the voxels, the number of cases that are repeated during training increases, similar to what was observed when changing the voxel size.

5.2. Automatic Classification of Architectural Objects Based on Point Cloud Data

To classify architectural objects, it is necessary to establish a training dataset. While a larger dataset generally leads to higher accuracy in results, the abundance of irrelevant data is unnecessary. To address this, we utilized the “Comprehensive 3D Indoor Data” collected from the AIHub (aihub.or.kr), operated by the Korea Artificial Intelligence Society for the Promotion of Information and Communications, to gather raw 3D point cloud data. This dataset is expected to be valuable for applications such as 3D object recognition in indoor spaces, training AI robot systems, and rendering 3D perspectives for architecture and interior design.

The Comprehensive 3D Indoor Data comprises a total of 731 indoor spaces, encompassing various types such as typical residences, single-person dwellings (studios, one-bedroom apartments, etc.), schools, libraries, hospitals, public offices, offices, factories, farms, art galleries, amusement parks, cafes, restaurants, accommodations, and more. Within each indoor space, there are a total of 120,000 spatial elements. These spatial elements include basic spatial components, open-close spatial components, furniture elements, electronic device elements, facility safety elements, and others, as seen in Figure 13.

Basic spatial components consist of elements like floors, walls, ceilings, columns, beams, stairs, and roofs. Furniture elements encompass items such as beds, tables, desks, chairs, sofas, storage cabinets, and others.

The training data utilized the ‘Comprehensive 3D Indoor Data’ from AIHub for semantic segmentation labeling. The labeling classes included cupboard, floor, wall, column, beam, window, door, table, chair, sofa, bookshelf, board, and miscellaneous, totaling 13 classes, following the format of the FCAF3D dataset. The aim of this study is the classification and generation of architectural objects, to utilize the generated model in the remodeling process. Consequently, the classified point cloud should align with real-world objects to be effectively employed in practical applications, as seen in Figure 14.

Using the FCAF3D network and the training model employed in this study, we conducted classification verification for the three architectural objects among the previously mentioned five (walls, columns, floors, roofs, and others) using architectural data other than the point cloud data used for training. In the case of this experimental building, there were no columns and roofs, and their boundaries were ambiguous, so they were excluded from the classification target. The following Figure 15a–d represent the results classified into each architectural object.

Figure 13 represents images classified into walls, floors, and others (chairs, desks), respectively. In Figure 13, when classifying walls, you can see that there is space left above and below the yellow Bounding Box. While the roof and columns of the target building were not clearly distinguished, since the dataset was structured with five categories during data collection (walls, floors, roofs, columns, others), you can see that the upper part corresponding to the roof and the lower part corresponding to the floor were not included in the walls.

Figure 13 shows the result of classifying the floor. However, because the dataset consists of point cloud data, it was unable to classify the thickness or volume of the floor. When extracting point cloud data, points are obtained based on factors like the time it takes for the laser to reflect. In this process, the surface of the floor can be obtained through laser reflection. However, the underside of the floor, depending on its thickness, cannot be captured, resulting in the classification of only the floor’s surface.

Figure 13 are the results of classifying other objects, excluding architectural objects, specifically chairs and desks. The “other” category encompasses various objects. Similar to the situation when finding a single wall within the “wall” category of Figure 13 represents the recognition and classification of individual objects within the “other” category, namely chairs and desks.

6. Conclusions

In this study, an architectural object recognition technology was developed to efficiently assist in the remodeling process of countless buildings constructed without blueprints. Aging buildings often undergo remodeling, a process that currently relies heavily on manual labor and is inefficient. In recent times, various smart technologies have been developed and integrated into different industries during the era of the Fourth Industrial Revolution. Smart devices like 3D scanners and drones are widely used for surveying purposes. Therefore, this research aimed to develop deep learning-based automatic object classification and recognition technology using point cloud data and smart technology to replace the traditional, inefficient approach.

Although various deep learning models and networks exist, this study utilized the FCAF3D network, which is voxel-based. By adjusting the voxel size and the number of blocks, the optimal training network environment was determined to optimize performance. Specifically, the voxel size was set to 1 cm, and training was conducted with four blocks. Using this approach, objects within 3D scanning data of real buildings were classified into walls, floors, roofs, and other objects.

The results of classifying architectural objects, including walls, floors, and roofs, displayed some ambiguity at object boundaries due to labeling using Bounding Boxes. Overlapping areas caused errors in object boundary classification. However, the classification of “other” objects within the architectural category clearly distinguished them from other objects, even when the boundaries were adjacent. This model demonstrated that 3D scanning and point cloud data can be automatically classified as “other” objects, reducing the manual deletion of these objects during the alignment process.

Furthermore, the automatic classification of architectural objects allows for the automatic differentiation between elements to be removed and those to be retained during the remodeling process.

While the research aimed to minimize loss of space by utilizing the minimum and maximum values of x, y, and z during the labeling process, there are limitations in obtaining precise numerical values since Bounding Boxes were used. However, the primary goal of this research was to develop a technology that more efficiently distinguishes objects within buildings, reducing labor costs and improving time efficiency during the architectural remodeling process. Based on this perspective, it can be concluded that the research aligned with its initial objectives.

In the future, the research may transition from using Bounding Boxes for labeling to using polygon forms almost to eliminate loss of space. As a result, the numerical output of classification results could be effectively utilized. Through subsequent research that integrates these outcomes with Building Information Modeling (BIM), it is expected that the classified object information will be valuable for tasks such as estimating the amount of construction waste and identifying non-demolished architectural frameworks. Continuous research in this direction has the potential to significantly enhance the efficiency of the architectural remodeling process.