Recent Advancements in AI-Enabled Smart Electronics Packaging for Structural Health Monitoring

Abstract

Real-time health monitoring of civil infrastructures is performed to maintain their structural integrity, sustainability, and serviceability for a longer time. With smart electronics and packaging technology, large amounts of complex monitoring data are generated, requiring sophisticated artificial intelligence (AI) techniques for their processing. With the advancement of technology, more complex AI models have been applied, from simple models to sophisticated deep learning (DL) models, for structural health monitoring (SHM). In this article, a comprehensive review is performed, primarily on the applications of AI models for SHM to maintain the sustainability of diverse civil infrastructures. Three smart data capturing methods of SHM, namely, camera-based, smartphone-based, and unmanned aerial vehicle (UAV)-based methods, are also discussed, having made the utilization of intelligent paradigms easier. UAV is found to be the most promising smart data acquisition technology, whereas convolution neural networks are the most impressive DL model reported for SHM. Furthermore, current challenges and future perspectives of AI-based SHM systems are also described separately. Moreover, the Internet of Things (IoT) and smart city concepts are explained to elaborate on the contributions of intelligent SHM systems. The integration of SHM with IoT and cloud-based computing is leading us towards the evolution of future smart cities.

1. Introduction

The electronics packaging industry is at the forefront of the artificial intelligence (AI) revolution. AI is widely applied in electronics and computer networks and is now being focused on the health monitoring of engineering structures. Civil constructions such as bridges, dams, multistory buildings, pipeline systems, etc., are very vulnerable to weathering impacts and changing loads [1,2]. Earthquakes, human-made excitation, wind, and weather conditions introduce unwanted vibrations in civil infrastructures and may trigger a catastrophic breakdown [1,2]. These structures are required to be supervised for repairing and strengthening the cracks, faults, etc., to control structural damage and maintain their sustainability [3,4]. To supervise civil structures’ health, structural health monitoring (SHM) methods have been implemented to maintain their service life and durability. SHM involves real-time sensory data that are recorded for monitoring the civil structures during loading and bad weather conditions [5,6,7]. The primary concern of SHM is to supervise the security and serviceability of structures such as bridges, buildings, flyovers, etc. SHM comprises four important steps, viz., data capture, system recognition, health evaluation, and decision for scheduling maintenance [8]. Conventionally, SHM utilizes vibration or strain-based methods to evaluate the in-service conditions of civil infrastructures in diverse climatic situations. These methods require the installation of contact or non-contact sensors on the target equipment for quick assessment of equipment health. Contact sensors including strain gauge, piezoelectric sensors, accelerometers, etc., are attached with the structural body to capture the dynamic response of civil structures efficiently [8,9,10,11]. Still, this type of sensor has its own advantages and limitations. Vibration sensors (such as pin/spring type, piezoelectric type, accelerometer, velocity sensor, and proximity sensor) are widely applied due to their low cost, the capability of capturing lower frequency responses and withstanding high temperatures, smaller size, ease of installation, and accurate measure of axial displacements. However, the installation of contact sensors is a labor-intensive, risky, and high-maintenance job [8]. Moreover, data captured through contact sensors are sparse, discrete, and have a low spatial resolution, which reduces the efficiency of SHM [12,13,14].

Wireless sensors are utilized to handle the challenges of contact sensors but have their own limitations [15,16,17]. Wireless sensors are to be installed in thousands of quantities on a civil structure, taking time to collect structural health monitoring data. Moreover, the collection of data from wireless sensors becomes a challenge as data broadcast is complicated, needs synchronization, and is power-demanding. To overcome the limitations of contact and wireless sensors, researchers have developed and implemented smart sensing technology for data capturing of SHM [18,19]. Real-time monitoring through smart sensors is a recent advancement, enabling the early detection and diagnosis of fractures or cracks for preventive maintenance of civil infrastructures. The preventive maintenance operations are carried out with optimized infrastructure resources resulting in smarter, intelligent, and more sustainable civil structures. These methods involve the application of sensors for patiently monitoring the dynamic reactions alongside the damage locations, and the evaluation of structural health over the lifespan [18,19]. Moreover, the captured information needs evaluation because of the multifaceted nature of captured data, interval management, and control utilization [8,9]. Damage detection for locating fractures’ position and capturing structural data is the most basic part of SHM [18,19].

In the 1990s, SHM went through significant changes because of smart sensors and data innovation. Exceptionally precise sensors, optical and remote systems, global positioning systems (GPS), and different advancements have added to the improvement of more accurate, cost-productive estimation and monitoring of civil structures [20]. As a result, the volume of informational indexes has expanded immensely, at the pace of a great many estimations for each sensor. These newer, better-off datasets present the capability to distinguish physical performance in uncommon manners, leading to worldwide interest in executing huge monitoring systems on civil architectures. The datasets captured and stored through SHM-based systems can also be considered as “big data” because of their data diversity, informational value, the velocity of data generation, volume, and data quality [21,22]. Thus, SHM has appeared as an innovation, driven by advanced technological proficiencies with comprehensible goals and possibilities [21,22].

Several researchers have supported the data-driven pattern recognition approach for the processing and analysis of the captured data [17,18,19,22]. In past years, researchers were doubtful about the accomplishment of SHM because it was generating an enormous quantity of data difficult to analyze properly. With the advancement in high computational power, storage capability, and machine learning (ML), the extraction of useful information from big SHM data has become possible in real time [20,21]. Therefore, knowledge about the changing health of civil structures over time is easy to monitor, record, and predict. The ever-diminishing costs of the computerized approach for handling, shipping, and putting away information has made SHM popular to the degree that a few analysts are utilizing the expression “information storm”. This has prompted a significant forward leap in design acknowledgment and encouraged the growth of computing in SHM [21]. The area of SHM research has expanded in diverse applications due to the technological advancement of sensors and artificial intelligence algorithms that are comprehensively and quantitatively investigated in this study. The distribution of research publications from 2006–2021 for SHM is shown in Figure 1.

In this paper, a comprehensive review of structural health monitoring is presented and consolidated from traditional methods to advance intelligent AI techniques. Additionally, three popular smart sensing techniques such as camera-based methods, smartphones, and unmanned aerial vehicles (UAV) are studied to provide a better understanding of data capturing techniques and their associated challenges. We also discuss the generalized steps utilized for ML applications to provide guidelines for quick understanding. The state of the art for SHM with AI applications is presented to compute research literature on the structural damage detection of buildings, bridges, and pipelines. Moreover, distinct sections are provided for addressing the current challenges and future perspectives of SHM technology. The advantages and disadvantages are also discussed. Various applications of ML techniques are conferred and supported with chronological tables containing information such as the name of researchers, year of publication, ML models, and associated references utilized for fault diagnosis. This will help researchers during the selection of suitable ML models for their work. The current research trends, associated challenges, and future scope of ML technique-based fault diagnosis have been discussed. The main objectives of this research are listed below.

• A comprehensive overview of ML applications in smart SHM electronics is elaborated.
• The state of the art of data-driven SHM is thoroughly studied and organized systematically.
• Separate tables are provided to illustrate diverse SHM methods chronologically for the reader’s convenience.
• Research trends of the past fifteen years are thoroughly studied and illustrated to identify the popularity of ML models for structural damage detection and localization.
• The implementation of smart sensing technologies for SHM is discussed in detail with the Internet of Things and smart city implications.
• The pros and cons of conventional and advanced SHM techniques are provided in tables that also highlight their important features.
• Current technological challenges and future research perspectives of AI algorithms in SHM are also discussed.

The purpose of this study is to consolidate new approaches for the applications of noteworthy AI methodologies in the past decade of structural engineering. However, the performance of the stated SHM-AI approaches depends heavily on the amount of data to be gathered via smart sensing-based monitoring devices. The implementation challenges of AI techniques to real-world power system scenarios are also discussed that may help field engineers to mitigate similar issues during their pragmatic applications. The paper is systematized as follows: Section 2 presents an overview of SHM, Section 3 provides an overview of AI for SHM, Section 4 explains the applications of AI models in SHM, and Section 5 discusses the challenges existing in ML implementation for SHM. Finally, Section 6 concludes the main results of this research and its future scope. Figure 1 shows the distribution of research publications from 2006–2021 for SHM utilizing AI techniques and smart sensors.

1.1. Research Methodology

The presented paper provides a comprehensive review of AI techniques in SHM along with the smart sensing technology that makes this possible. The overall discussion is based on 207 articles in related fields from the year 2006 onwards, of which 181 are journal papers and 26 are conference papers while, the remainder includes books or digital books. The papers included are directly or indirectly related to SHM, deep learning (DL), ML, and smart sensors, viz., UAV, smartphones, and cameras. Journal articles were the first preference for inclusion; however, relevant conference papers are included. Several academic repositories such as ScienceDirect, IEEE, Taylor and Francis, Sage, Web of Science, and Scopus were searched to collect the relevant research works. The keywords such as “structural health monitoring”, “machine learning”, deep learning civil structural health monitoring”, “data mining health monitoring”, “artificial intelligence health monitoring”, etc., were searched in the abovementioned databases to recognize the research works related with AI application in SHM. The search periods were set from 2005 to 2021, resulting in the identification of 501 works. The screening criteria for identified researcher works was SHM-related implementation of data analytics, data mining, ML, and DL. Initially, titles, keywords, and abstracts were studied thoroughly then the papers were read and analyzed if the abstract was found suitable. All the selected papers are close to the research objective of this article. Finally, 230 articles were selected and utilized for this review work. Figure 2 shows a pictorial representation of the addition and elimination criteria of papers to validate the review of prognostics.

The pie chart presented in Figure 3a depicts the percentage share of research papers included from various publishers of those journals separately. Moreover, the histogram shown in Figure 3b gives the impact factor of journal papers, which further validates the value of the papers.

The contribution of this review work can be summarized as given below.

• A comprehensive review was performed on advanced smart data acquisition methods for SHM.
• AI applications in SHM were studied to deliver a broad review of the existing technologies and advancements.
• We provide a brief description of diverse challenges prevailing in the SHM domain.
• We offer insights about future research directions and challenges in the application of ML for SHM.
• We provide a chronological arrangement of recently published works for diverse sensor-based techniques for reader ease.
• Efforts were made to illustrate various methodologies pictorially.

Data Diversity and Databases Utilized for SHM

Data are the most important aspect of artificial intelligence models, whether they are applied for SHM or any other engineering field. Due to advancements in smart sensing technology, large amounts of real-time and experimental research data are available in the public domain that is useful for testing newer intelligent paradigms. Three are three common data formats that are widely produced during SHM operations, namely, time-series vibrational data, 2D pictures, and 3D point cloud data. Time series data are the most common form of data types primarily recorded from vibrational sensors installed for damage detection. The real-time measurements can be captured through single or multiple time series data format depending on the number of sensors placed on the structure under surveillance. The next data format of SHM exists in the form of 2D pictures of cracks, factures, etc., which are processed through image processing techniques. The third data format which prevails in SHM is point cloud data generated from laser light detection and ranging devices such as terrestrial laser scanners (TLS), etc., and require sophisticated simulation software for its processing. Point cloud data are 3D data generated through the measurement of laser reflection from the object’s surface. Later, these data points are converted into mesh models, surface models, or computer-aided design (CAD) models through the process of surface reconstruction. There are several online databases available for SHM such as “Los Alamos National Laboratory, U.S.A”, “IASC-ASCE Task Group on Structural Health Monitoring”, etc. The list of these open-source databases is provided in Appendix A,

Table A1. Vibrational and images datasets available for research purposes.

S. No.	Source	Data Types	Dataset Description
1	Los Alamos National Laboratory	Diverse experimental data types	Alamosa Canyon Bridge Data, I-40 Bridge Data, UC-Irvine Bridge Column Data, Sheraton Hotel (Universal City, CA) Ambient Vibration Data, 8-DOF System Data, Bookshelf Frame Structure—DSS 2000, Bookshelf Frame Structure and 4-DOF Nonlinearity data set
2.	Laboratory of Infrastructure Science and Sustainability	Diverse data types	Camera-based motion magnification, vibrational data of building, etc.
3.	IASC-ASCE Task Group on Structural Health Monitoring	Simulated data	SHM of civil structure using simulated vibration data
4	National Center for Research on Earthquake Engineering	Diverse vibrational and images data related to earthquakes	Structural vibrational frame test data in diverse earthquake scenarios, 5000 data images, etc.
5	IEEE Dataport	Vibrational data and LIDAR data	Multiple LIDAR datasets such as aerial warehouse data, indoor experimental data collected from two 3D sensors, etc.
6	Maguire et al. [231]	SDNET2018	56,000 2D images of deck, wall, and pavement of a bridge
7	Hoskere et al. [232]	2D image data	Pictures of building, walls, etc.; laboratory experimental results
8	Avci et al. [65]	Vibrational data	330 signals each containing 245,760 samples
9	Beckman et al. [233]	Vision-based data	444 concrete spalling images (853 × 1440 pixels)
10	Bao et al. [234]	Vision data	333,792 acceleration signals
11	Dung and Anh [235]	2D image data	An open database with 40,000 data samples of concrete fractures with 227 × 227 pixel pictures
12	Lin et al. [236]	Vibrational data	459 datasets captured from nine nodes present in 1024 × 9 matrices
13	Gulgec et al. [68]	2D image data	30,000 unhealthy and 30,000 undamaged tension distribution matrices in 28 × 56 measurement
14	Tung et al. [237]	Vibrational data	10,014 responses of a long hanging bridge cable having two output channels with 100 × 100 resolution
15	Nahata et al. [238]	2D image data	224 × 224 × 3 pixels containing 1200 RGB image data
16	Ni et al. [239]	Deep learning image repository	Images (RGB 224 × 224 pixels) for GoogLeNet with 60,000 pictures
17	Duarte et al. [240]	2D image data	12,973 pictures of the satellite and an airborne vehicle (224 × 224 resolution)
18	Kim et al. [186]	2D image data	3186 pictures of cracks and intact surfaces (227 × 227 pixels)
19	Chen [241]	Vision data	147,344 cracks and 149,460 non-cracks (120 × 120 patches)
20	Sharma et al. [242]	2D image data	15,600 factures and non-factures (28 × 28 RGB patches)
21	Pathirage [243]	Vibrational data	Modal information of 10,300 fractures (7 arrays)
22	Wang et al. [244]	2D image data	500 pictures (500 × 500 pixels)
23	Dorafshan et al. [245]	2D image data	9011 (227 × 227 pixel) images of lab-made bridge decks
25	Jang et al. [246]	2D image data with vision data	20,000 hybrid images by merging vision and infrared thermography of concrete factures (224 × 224 pixel)
27	Yu et al. [247]	Vibrational data	1900 clusters of 5 × 2832 matrices
28	Ye et al. [248]	2D image data	14,000 fracture pictures (80 × 80 resolution)
29	Modarres et al. [249]	2D image data	Data of 2400 concrete fractures, sandwich panels with a resolution of 96 × 96 pixels
30	Zhang et al. [250]	Vibrational data	8595, 14,465, and 4800 vibrational signal (9 Ch. × 10,000)
31	Xu et al. [251]	2D image data	2400 pictures (640 × 640 resolution), concrete fractures, spalling, rebar exposure, and buckling
32	Yang et al. [118]	2D image data	800 images of cracks (224 × 224-pixel resolution)
33	Abdeljaber et al. [65]	Vibrational data	749 × 12 vectors of vibration signals (128 × 1 dimension)
34	Atha and Jahanshahi [252]	2D image data	67,187 images of corrosion (128 × 128 resolution)
35	Zhang et al. [253]	2D image data	300 images with 224 × 224 pixels
36	Silva et al. [254]	2D image data	3500 sample images of 256 × 256 pixels
37	Kumar et al. [255]	2D image data	12,000 pictures of fractures, extreme cracks, intrusions, scaling, deposits, corrosion in pipelines (256 × 256 dimension)

. Several individuals have also provided their open research data for SHM. Information about individual research databases is also included in the Appendix A, Table A1.

2. Overview of Structural Health Monitoring

The concept of measuring basic vibrational responses of civil structures exists for the detection and location of damage over a long period of time. This consideration in common mechanical and aviation assemblies to comprehend and recognize the damage is prevalent in the basic design field [23]. SHM employing the vibrational inspection technique contains five levels, namely, discovery, localization, grouping, appraisal, and forecast [24]. Direct and non-straight sorts are the two significant arrangements implied for dynamic damage. A straight flexible structure will occur as the equivalent even after damage, where the modular characteristics and varieties because of mathematical or materialistic changes can be exhibited utilizing a direct condition [25]. Non-direct damage happens when a straight versatile structure deviates into a non-direct way afterward the event of damage. For example, the progress of a crack that may consequently expose and close under vibrations in its ordinary working conditions is a suitable model for nonlinear damage.

A solid damage recognition method is pertinent to both kinds of damages as a rule. As referenced previously, the cycle of SHM includes various developments. Figure 4 depicts a schematic workflow of SHM for damage detection.

From the start, the structural framework is checked after some time, using several sensors and raw data that are gathered dependent on intermittent examples of estimations of dynamic reactions obtained from similar sensors. Extraction of the information is the next phase, where the tops that can achieve damage are removed from these experimental evaluations [26]. An inspection is executed on extracted damage gentle highlights to survey the present circumstances and soundness of the supporting framework [15]. Should there be an existence of long-haul dynamic checking situations, the yield of such a measurable cycle is refreshed routinely, to acquire data that validate the limit of the structure easily when it is exposed to maturation and crumbling because of different natural conditions, or when the structure experiences adverse effects because of events such as tremors or overwhelming loading [27].

SHM is a means to approve the real-world reliability of civil structures. The breakdown of the North Carolina Bridge in the US is an event that pushed engineers to concentrate on dynamic fitness observing procedures. Additionally, the progression in remote sensor systems has impacted SHM innovation and encouraged the remote broadcast of the observed boundaries, for the most part highlighting the distant access of the SHM frameworks [28,29]. Figure 5 shows a depiction of different conventional techniques used for leakage detection. Figure 6 shows the RGIPT plant demonstrating pipe leakage and structural damages in water pipelines.

Table A2. Diverse techniques utilized for detection of corrosion in transmission pipeline to maintain structural integrity [256].

S. No	Method Name	Description	Applicability	Results	Use for Water-Based or Oil-Based Conduit	Comments for Future Prediction of Damages
1	Corrosion Coupon	Coupon is placed within the working material, thus invasive	Can be applicable for pipe/reactor of any shape or size	No precision position and time of leak/corrosion	Water-based system	Difficult to predict any future damage location
2	Electrical Resistance Probe	Invasive probe works as a real-time corrosion coupon	Can be applicable for pipe/reactor of any shape or size	No precise positioning but time and extent of corrosion or mass loss can be determined	Suitable for oil and water-based system	Real-time data may be utilized to detect the future damage or probable future leaks
3	Electrochemical Sensors	In situ electrochemical corrosion rate determination	Can be applicable for pipe/reactor of any shape or size	No precise positioning but time and extent of corrosion can be determined	Works better for ion conducting electrolytes	Difficult to predict any future damage location
4	Ultrasonic (Acoustic) Testing Sensor	Ultrasonic probes are placed inside the pipe to detect pipe thickness, flow change or loss	Can be applicable for pipe/reactor of any shape or size	Good precision; real-time positioning is possible; not suitable for very small leaks or structural damages	Suitable for oil and water-based system	Real-time data can be utilized to detect the future damage or fault location
5	Magnetic Flux Leakage Method	Invasive technique for detection of damage in structure by comparing magnetic flux lines	Can be applicable for pipe of any shape or size	Cannot precisely locate the position of structural damage	Oil/water-based system	Difficult to predict damage location
6	Point OFS for Corrosion	Optical corrosion coupon using optical spectrum from its position inside the pipe	Can be applicable for pipe of any shape or size	No precise positioning but incidence and extent of corrosion can be determined	Suitable for oil/water-based system	Difficult to predict damage location
7	Quasi-Distributed OFS for Corrosion	It uses FBG based external point sensors to determine change in temperature and strain; the pressure wave generated transmits in both the directions from point of leakage, where the pressure sensors detect the leakage point my analyzing pressure wave	Very useful to determine the corrosion in pipeline and wellbore in real time	Precise point and time of leakage can be determined using this technique of negative pressure wave (NPW)	Suitable for oil or water-based system, it can detect gas leaks	Can be useful for predicting future leak or damage
8	Distributed OFS for Physical Sensing	Parameters of corrosion and leaks are determined by monitoring pressure and temperature change due to leak; optical fibers are winded over the pipe to detect the leak	The technique is also useful for determination of efficient flow of crude in pipes and impacts in flow due to corrosion; estimates corrosion and structural changes	The leak can be determined precisely and in real time	Can work for conduits carrying oil, waters, and gas	The technology can be extended to determine corrosion or damages in pipe
9	Distributed OFS for Chemical Sensing	Optical fibers with chemical coating and air holes is activated over pipe core or cladding, can be applied to check the external or internal health of piping structure	Multi-sensor OFS are designed and utilized to determine leaks of gases of different types and nature of environments the conduits are exposed to	Precise determination of leaks and damages are possible in real time	Can work for conduits carrying oil, waters, and gas for leak detection	It gives early signs of corrosion; it is the best method to predict the damage or leak
10	SCADA and CMS	Acoustic emission, optic fiber, thermographic, photogrammetric techniques and other are used to remotely collect and monitor the surface of structure	These can be used to detect damages in pipelines and other civil infrastructures	Cracks can be easily monitored; very fine cracks may not be detected through real-time data	Can work for conduits carrying oil, waters, and gas	Monitoring external conditions may not hint always any impending danger
11	UAV-Based Technique	Multi-sensor (thermal, laser, sonic, spectroscopic, photogrammetric) remote sensing of crack, and structural deformations using UAV platform	Determine the surface damages to any infrastructure of oil and gas industry	Laser UAV can detect fine damages if scanning is performed from close proximity; data are required to be analyzed to determine the leaks	Can work over oil, water, gas conduits or any other infrastructure	The damages existing in pipelines or infrastructure may be extrapolated to determine the future source of leak or gas emissions

contains diverse techniques utilized for the detection of corrosion in transmission pipelines to maintain structural integrity.

Indifference to city-wide appropriation frameworks, pipelines, and building frameworks can be influenced by minor leakages that can be recognized simply after extensive or complex activities, such as checking estimated asset permits at the meter during times of no asset use [30]. They may stay unnoticed for quite a long time or months, subsequently generating an enormous amount of waste or even damage. The insights detailed in the Water Sense Project [31] display that private water spills in the United States waste around 3.78 trillion liters of water per year. Numerous inadequacies influence the progress of legitimate applications for observing water and gaseous petrol matrices, as explained in the thorough study by Fagiani et al. [32], regardless of the accessibility of cutting-edge metering frameworks [33,34] and sensor deficiency in structures. In particular, a wide arrangement of artificial intelligence and computational approaches zero in on the satisfaction of the shrewd household and the keen matrix ideal models [35,36]. Innovative exposure methods raised profound interest in numerous applicable fields, and a huge quantity of writing has been created to this point [37,38]. Among these, non-meddling strategies [39,40] have been considered quite compelling since they do not need extra instruments, other than the water meter, to find the spillage. In addition, ample information about spillage/issue identification in mechanical conditions is accessible in the literature. Generally, oil and petroleum gas pipelines depend on high inspection rates as well as numerous detecting sensors arranged along the pipeline [41,42].

SHM can bolster the procedure of executing a damage recognition approach for civic structures [43]. Damage is the change in physical and statistical characteristics that shifts towards the worst conditions [43]. In other words, the damage is whatever affects structural performance. Structural health monitoring involves routine indicators using an image or precise data and an investigative model together with engineering knowledge. The final consequence is the health profile of the structure through the performance profile, which makes it possible to reliably predict the future health of the structure [44]. Figure 7 shows the placement of sensors for the SHM of bridge structures.

Damage to the structure is nonlinear and hysteretic. Natural elements (e.g., temperature) can change the common recurrence of structures with no damage to the structure [45]. Similarly, the between-story floats, if not determined by coordinating the band-pass separating increasing speed information, are certifiably not dependable damage markers, as the mistakes created by the commotion in the records are dramatically enhanced during such a mix [46,47]. A smart algorithm is created to process real-time data at the same time and this algorithm also gives a very precise location of the damage [48]. Therefore, real-time data processing gives a smart solution for damage detection as well as sheath monitoring of the structures [49]. A systematic review was provided for dynamic damage detection using the subspace technique for civil constructions [50].

The health profile, for both diagnosis and prognosis, typically depends on SHM. These techniques are generally mentioned as model updating or device recognition and consist of searching the parametric real-world models that optimally suit the structural data produced by the sensors to gather information that cannot be calculated on-site directly [51,52]. As referenced previously, the cycle of SHM includes various advances. From the start, the framework is checked over a long time, utilizing many sensors, and perceptions are guessed dependent on intermittent examples of estimations of dynamic reactions acquired from similar sensors [53]. Extraction of the information is the next step, whereas the data, which represent damage, are separated from the noticed estimations [54]. Furthermore, a measurable examination is executed on this removed damage-related information to survey the current conditions and wellbeing of the basic framework [55]. If there should arise an occurrence of long-haul dynamic situations, the yield of such a factual cycle is refreshed routinely, to acquire data that validate the limit of the structure easily when it is exposed to maturing and crumbling coming due to different natural conditions [56,57,58]. These sensors are simpler, user-friendly, and permit quicker installation to capture monitoring data that has high-resolution and spatial structural data. These sensors are also small-work concentrated as well as profoundly profitable. Here, we present a comprehensive review of the existing literature on current sensors-based technologies, namely, cameras, unmanned aerial vehicles, cell phones, and acoustic sensors [58].

2.1. Overview of Artificial Intelligence

Artificial intelligent (AI) models such as ML and DL are extensively utilized in the domain of SHM for several forms of damage, leakage, and health monitoring purposes [59,60]. AI is proven to be a cost-effective alternative to traditional modeling methods. AI is a computer science subject that focuses on the production of human intelligence machines and software. Many problems in civil and structural engineering are affected by uncertainties that cannot be addressed by traditional techniques. In solving these difficulties, the application of AI can help. In addition, AI-based solutions for the determination of engineering design parameters are practicable when testing is impossible, leading to significant savings of time and effort in tests. Moreover, AI may accelerate the process of decision making, decrease error rates, and enhance processing efficiency.

ML is often considered to be synonymous with AI, yet it covers other smart concerns such as grouping, groupings, computer vision, etc. ML debates computers with sophisticated human behavior, whereas AI denotes a machine’s potential to mimic human cognitive processes. AI is a wider phrase and concept that includes applications where a machine imitates “cognitive” capabilities associated with humans and their brains, e.g., “learning patterns” and “finding solutions of problems”. ML is a subset of AI which includes “traditional” paradigms for diverse tasks, such as classification, estimation, or clustering. ML models must be trained on a sufficient quantity of data to learn hidden patterns and also to achieve accurate predictions. Data mining/science is a wide field that tries to uncover essential valuable information in vast volumes of data. Data mining practices are utilized to find unknown qualities in domains where knowledge is sparse. The datasets are classified as “big data” only if it has five V’s—volume, velocity, value, velocity, and variety—as its qualities. Big data relate to large or complex data sets that employ standard data analytics techniques which are essential to extract hidden information. ML may be used to develop an AI subfield to detect the hidden patterns and emphasizes the prediction based on recognized attributes that have been learned from the training datasets. DL is another subset of ML, a method that works on gaining knowledge about the data’s structures, their properties, and hidden patterns. DL is developed to handle big data efficiently with machine vision-related problems. It utilizes multi-layered deep neural network architecture for learning the hidden patterns of training data and has distinct inbuilt capabilities that are absent in conventional neural networks such as feature extraction, big data handling, machine vision, etc.

In the subject of structural engineering, uncertainty affects several aspects of architecture, analysis, status monitoring, project management, and decision making. In certain aspects, the issue of structural engineering is unavoidable. In seismic design, for example, earthquake requirements are not understood precisely. In structural patient monitoring, there are mistakes in the amplitude of the input stimulation, noise measurement, and spatial density. Models designed to predict and characterize structural responses can contain large uncertainties. Geotechnical information is obtained by the use of limited information or laboratory testing data for the high level of uncertainty existing in structural base applications. All these glitches may be simulated and regarded as uncertainties [61]. However, AI can address such uncertainties effectively. Updates of the finite element model were utilized to address problems of uncertainty for instance within the context of system diagnosis of damage [62]. The updating of the model may be used to identify seven physical parameters for which value changes are utilized to signal damage (e.g., the rigidity of a structural component). However, such reductions can only be caused by statistical uncertainty. Thus, the uncertainty of the estimate must be calculated to see if a parameter drop can be caused by real damage. In addition, the application of AI techniques may save time and money and enhance computing efficiency in various structural engineering activities. Figure 8 shows (a) different advanced data-driven techniques utilized for SHM and (b) generalized workflow for Azure Machine Learning proposed by David Chappell [63].

2.1.1. Deep Learning in SHM

Recently, the utilization of DL has increased for structural engineering purposes. The implementation of DL (e.g., Convolutional Neural Networks (CNNs)) for damage detection is very innovative. CNNs learn and extract through provided features to achieve optimal features and classification. This makes a very efficient image-recognition workflow convenient to design for 2D signals such as images, video frames, etc. CNNs are classified and employed as vision-based SHM techniques, which record images in different structural states.

Table 1. The evaluation of ML vs. DL approaches for SHM applications.

	Machine Learning (ML)	Deep Learning (DL)
Merits	This can be applied in conventional and data-driven systems of SHM.	DL can be applied with conventional systems. Mainly applicable for vision-based SHM systems.
	ML can be combined with Internet of Things (IoTs) for smaller implementation.	DL can easily handle big data conditions.
	Relevant for smaller optimization problems with proper model parameters tuning.	Applicable for large optimization problems without tuning of parameters.
	Computationally efficient with less data.	Computationally efficient with high data volumes.
Demerits	Not suitable for newer SHM systems for video processing.	Not suitable for conventional SHM systems.
	The data structure of ML is different than CNNs.	Large amounts of data are required for effective performance.
	Normally, low computation cost when compared to DL.	Higher computational cost than ML.

shows the comparison between ML and DL properties for SHM applications.

Sarkar et al. [64] first used CNNs to classify crack damage of composite materials in structural engineering. Abdeljaber et al. [65,66] introduced a one-dimensional CNN approach with vibration-based damage detection. They verified that the technique could be directly learned from the measured acceleration statistics, resulting in a proper approach to civil structure health monitoring. However, the proposed system required many data capture sessions to generate the training data, particularly for large civil structures, which is a limitation. They proposed a method for identifying nonparametric damage through CNNs which required two training data capture sessions to overcome the above-mentioned drawback [66]. They demonstrated the efficacy of the SHM system in the detection of actual damage. Cha et al. [67] introduced a DL model for sensing concrete cracks in tunnels without computational defects. A comparative study was also carried out to understand how concrete cracks were detected robustly as compared to traditional image detection methods in the proposed DL damage assessment approach. To find out the unidentified association between captured data and patterns of damage, Gulgec et al. [68] proposed the method to identify the structural damages using CNNs. The applicability of DL in a ten-bar planar truss for structural analysis was further examined by Lee et al. [69]. It was shown that conventional techniques were less efficient than CNNs. All these studies suggest effective tools for SHM using DL/CNNs architectures and established these frameworks as feasible procedures for newer vision-based SHM systems. Figure 9 shows the utilization of CNNs for crack detection in buildings adopted from [70].

2.1.2. Machine Learning-Based Damage Detection for SHM

Initially, Bayesian probabilistic models were applied with an artificial neural network (ANN) for small structural monitoring, damage localization, and determination of its severity [71]. The dynamic response is utilized for SHM using ANN along with signal anomaly index which represents a deviation in the shape of the frequency response function [71]. Additionally, the effect of noise was also studied on the ANN outcomes. ANN and model variables were employed for the SHM to monitor a steel frame five-story building. The mode frequencies were taken as input variables and stiffness as a response. White noise was added to understand its effect [72]. Diverse activation functions of ANN were tested with IASC-ASCE standard structure data. Model parameters were also compared with the Ritz vector. It was found that model parameters were more suitable for the training of ANN for SHM applications with the presence of an adequate amount of noise within the captured training data [73]. A hybrid neuro-wavelet technique was employed for damage recognition in SHM [74]. The input data comprised the Gaussian noise that was intentionally added to generate noisy training data to study its impact. A two-level ANN was applied for the estimation of unmeasured mode shape, severity, and localization of structural damage. A hybrid combining ANN and a genetic algorithm was proposed for damage detection in [75].

A nonparametric damage recognition paradigm was proposed based on a self-organizing map to extract damage indicators without utilizing modal frequency data. Later, feature extraction was implemented to detect structural damage using accelerometer data. K-nearest neighbor (K-NN) and support vector machine (SVM) were applied for the damage detection of rotary machines [76]. Later, variations of K-NN were also reported for damage detection [77]. The residual error was utilized for damage recognition through an autoregressive model with the acceleration time series [78]. SVM was employed for SHM with velocity, acceleration, and displacement as input features [79,80]. Further, CNNs were applied for damage detection using accelerometer data [79,80]. CNNs were also applied on a wireless sensors network with raw vibration signals which included 1D CNNs for every wireless node [81,82]. A separate study was performed utilizing a smartphone to detect seismic damage [83,84]. The smartphone accelerometers were applied to determine the structural displacement of multistory buildings due to shaking induced by earthquakes [85]. These smartphones contained low-quality sensors; thus, a high pass filter was used for noise cancellation [85], along with a low-quality accelerometer to learn the impact of the noisy dataset on the enactment of machine learning models for SHM applications [86]. A detailed review of damage recognition was provided on the bridge structures to describe the diverse levels of damage. It elaborated four levels, namely, (a) quick detection of damages, (b) localization of identified damage, (c) assessment of damage severity, and (d) prediction of existing structures life [87]. It also identified three limitations of model-based damage detection, namely, changing environmental conditions, improper application of intelligent data-driven models, and too much confidence over sensitive features of captured data for the bridge structure [87].

2.1.3. Data-Driven Pipeline Leakage Detection

In the industry, the transportation of a bulk number of fluids is performed through pipelines. These pipelines are usually damaged with time because of seasonal weathering conditions. Thus, continuous monitoring of pipelines is performed to maintain their structural integrity and repair damage.

Figure 10 shows pipeline leakage detection using LIDAR technology. The initial real-time model was proposed to measure the short-lived behavior of fluid to identify the leaks using adaptive state observers, mathematical dynamic models, and correlation identification techniques [80]. Further, fuzzy logic, ANN, and genetic algorithms were applied to develop an expert system for the identification of seepage in oil pipelines [88]. In another work, SVM was utilized for leakage detection using noisy training data [89,90]. The method of hierarchical seepage recognition and localization in sensor networks was anticipated for the monitoring of the natural gas pipeline [91]. The pipeline system was modeled in EPANET software to generate the training data for ML models, viz., ANN and SVM [92]. They also compared the performance of ANN and SVM for the detection, localization, and estimation of the size of pipe leakages [92]. A novel workflow was also proposed for the seepage detection of water/natural gas grids. It utilized the Sequential Feature Selection paradigm (SFSA) for the feature extraction and three machine learning techniques, namely, Gaussian Mixture Models (GMM), One-Class Support Vector Machine (OCSVM), and Hidden Markov Model (HMM), for leakage detection. The backpropagation ANN for the water resources management of Rajasthan illustrated the potential of several ML models [93,94]. Another work compared ANN, SVM, Logistic Regression, and Random Forest (RF) algorithms for monitoring the water distribution network of Italy [95,96].

A stacking ensemble was applied to combine the results of multilinear regression, ANN, SVM, and RF for the determination of pipe performance. This work also reported that there was a 35.7% increase in the prediction accuracy with a 13.6% reduction in error when compared to the single supervised algorithm [97]. The existing technologies were reviewed for the identification and localization of leakages to use in water pipeline systems using wireless sensor networks (WSNs) [98]. The fusion of 1D CNN–SVM was implemented for the leakage detection and graph-based method for the localization of leakage fault [92,99]. Later, a comparative study reached a detection accurateness of 99.3% and localization error was less than 3 m when applied to the actual water circulation network [100]. Intrinsic mode function, principal component analysis, and approximate entropy were implemented to extract important features from data collected from 4G wireless sensors networks installed at water distribution systems [101]. These extracted features were utilized to train the SVM model to spot seepages in the water pipeline system. Wavelet decomposition was employed to extract important features and ensemble methods for the identification of seepages in water circulation networks through pressure analysis [102]. A review was published about pipe leakage recognition systems and data fusion [103]. A deep learning technique was implemented for SHM utilizing exceptionally compressed data [104].

Table A3. Recent publications on data-driven methods applied in SHM for diverse purposes.

Authors	Year	Type of SHM	Techniques	Reference
Mosalam et al.	2019	Civil infrastructure	Deep learning	[54]
An et al.	2019	Bridge and truss structure	Deep learning	[55]
Baroudi et al.	2019	Oil and gas infrastructure and pipelines	Data fusion techniques	[90]
Aalsalem et al.	2018	Oil and gas pipelines	ANNs	[57]
Abdeljaber et al.	2018	Large framed structure	1D CNNs	[66]
Vitola et al.	2017	Large civil structure	SVM and K-NN	[49]
Abdeljaber et al.	2017	Steel framed structures	1D CNN	[65]
Goyal et al.	2017	Water pipelines	ANN	[82]
Shi et al.	2017	Pipeline structures	ML, SVM	[85]
Santos et al.	2016	Bridge and cable structure	ANN and Clustering techniques	[47]
Vitola et al.	2016	Large civil structure	PCA, K-NN, SVM, Boosted tree, Bagged tree, Subspace K-NN, Subspace Discriminant, and RUSBoosted Trees	[208]
Santos et al.	2015	Bridge structure	Unsupervised detection	[47]
Fagiani et al.	2015	Gas pipelines and water grid structure	SFSA, GMM, HMM, and OCSVM	[32]
Nasir et al.	2014	Oil and gas pipelines	ANN and SVM techniques	[80]
Khaleghi et al.	2013	Building structures	Multi-sensors data fusion	[56]
Wan et al.	2012	Natural gas pipeline	SVM	[79]
Jayawardhana et al.	2011	R.C.C. Slab	Autocorrelation Function-Cross Correlation Function and Auto-Regressive (AR) time series model	[48]
Laurentys et al.	2011	Pipeline structures	ANN	[76]
Glaser et al.	2008	Tall structures	Microsensors wireless network technology	[52]
González et al.	2008	Earthquake-resistant building	ANN	[59]
Farrar et al.	2007	Civil and mechanical structures	Statistical pattern recognition	[51]
Yuen et al.	2006	Symmetrical large civil structure	ANN along with feature extraction	[58]
Chen et al.	2004	Water pipelines	SVM	[77]

shows recent publications on various data-driven methods applied for SHM (in Appendix A).

3. Next-Generation Smart Electronic Packaging Solutions

Electronic contact sensors are attached to the structural body to capture the dynamic response of civil structures [8]. However, the installation of contact sensors is practically labor-intensive, risky, and high-maintenance [8]. Data captured through contact sensors are sparse, discrete, and have a low spatial resolution, which reduces the efficiency of SHM [8,9,10,11]. Wireless sensors are utilized to handle the challenges of contact sensors but are reported to have their own limitations. Wireless sensors must be installed in thousands of quantities on a civil infrastructure and it takes time to collect the SHM data [8,12,13,14]. Moreover, the data acquisition of wireless sensors becomes a thought-provoking job due to the complication of data transmission, synchronization, and power consumption [8,11,13,14]. The non-contact sensors are reported to be costly and less effective with varying climate conditions and distance measurements. Longer distance measurement through non-contact sensors requires a higher intensity laser light for its measurements [8,11,12,14]. This is hazardous for structures when data are collected for SHM. The above difficulties of ordinary non-contact sensors have been removed with the ongoing advancement of diverse smart sensors and electronic packaging inventions that are coordinated with visual and versatile checking frameworks [105,106,107,108]. Smart sensors capture data from their surrounding environment and utilize their computational power to evaluate predefined functions when particular signals or inputs are perceived. They pre-process the captured raw data typically using a Digital Motion Processor before transmitting them to cloud computing platform for further analysis. They contain analog filters, transducers, excitation control, battery power source, amplifiers, and inbuilt software functions for data digitization, onboard pre-processing, and transmission. Further, smart sensors are integrated with Internet of Things (IoT) gateways to cloud computation platforms. IoT networks also enable real-time remote monitoring of connected systems and are often termed smart-based monitoring devices in Industry 4.0. These smart detecting strategies incorporate advanced and ultra-high-speed cameras, UAVs, cell phones, and portable (mechanical) sensors [109,110,111,112]. Figure 11 shows a diagram of several smart next-generation sensing technologies applied for SHM. We discuss the important techniques in this section.

3.1. Camera Method

Low-cost vision sensors have been developed that assist the health monitoring of civil structures remotely. Nowadays, digital single-lens reflex (DSLR) and a high-speed camera can be utilized for the data acquisition of SHM data. Digital single-lens reflex cameras and exceptionally advanced cameras have been used for information securing of small buildings [8]. Cameras are differentiated using pixels, data transfer capacity, and picture adjustment. The current camera methods are different in variety and range from digital image correlation (DIC) to motion magnification (MM). There are four primary steps involved: (a) camera adjustment, (b) image acquisition and refinement, (c) displacement field measurements, and (d) damage detection [8]. DIC fails to detect a low amplitude level of motions in civil structures due to high frequency. Hence, motion magnification technology has been developed to measure the displacement shape of vibrating structures. Up to a particular frequency, DIC and MM in combination can measure minor displacements during higher frequency vibrations [10,11,12]. A non-contact vision camera system was applied for multiple locations’ dislocation monitoring in a cable-stayed footbridge [113]. The camera method mainly contains four stages with camera adjustment in a steel box brace and captures data by taking pictures using the user camera. The impact of pictures and component dimensions are concentrated to comprehend the strength of the suggested strategy [113]. Sometimes, for structures and scaffolds, multipoint dislodging observing is basic, and the advancement of a lower price camera-based method is fundamental for its usage in industries. The vision-based framework was provided for estimating the dislocations of large structures along with concurrent adaptive calibration and full-motion assessment [114]. The real-time identifying of native damages was proposed in civil structures using robust principal component analysis [115].

An update for the advanced high-speed correlation framework was developed to assess the parameters of two thin steel samples [116]. Q-450 Dante dynamic’s cameras were used with a testing recurrence of 2000 fps, and the subsequent documents were spared as various leveled information design through Intra4D [116]. A mechanized instrument called Modan3D was created to process pictures directly from available data format files to recognize the model. The 3D digital image correlation technique was investigated for the identification, localization, and quantification of damage in an aluminum cantilever beam using a stereo camera pair to capture the object’s surface [109]. Dynamic load conditions were also applied to capture the beam response on the first three resonant frequencies. This technique was found suitable for large cracks and faults in the beam structure [117].

A novel non-contact vision sensor was developed based on two advanced template matching techniques: the unsampled cross-correlation (UCC) and the orientation code matching (OCM). These templates are utilized for synchronized recording of structural dislocations at multiple points through a single camera [118]. Substantial benefits of this anticipated vision sensor include its low cost and flexibility to extract defect information at any point from a single vision-based measurement [118]. With the decent information procurement capacity of a camera, the requirement to build up a coordinated picture and video investigation application was also examined [119].

In another innovative procedure, a video picture handling method was created to address difficulties related to vision sensors [120]. The basic challenges, namely, restricted lighting, multipoint relocation, and camera vibration related to vision sensors, were tested. A low-cost camera vision scheme for capturing multipoint dislocations on a cheaper consumer camera was investigated for video capturing and its processing. This entire system was validated on a cable-stayed footbridge for deformation and vibration data recording under pedestrian loading [121]. Practically identical research work was performed utilizing 2D digital image correction and fiber Bragg grinding to examine its adequacy in estimating the relocation of barrier spans in different imperatives.

A review of the state of the art related to a vision-based scheme is provided for the displacement measurements of civil structures [98]. The processing of captured video files of vision-based systems is organized in this work into three components: structural displacement calculation, target tracking, and camera calibration, all with their limitations and advantages. The deformations in bridge assembly and cable vibrations were also investigated properly. Significant existing research gaps were also investigated for robust tracking approaches, non-contact sensing methods, and data capturing accuracy in real field situations [98]. As introduced above, the utilization of cameras and important picture handling calculations of the structure are among a wide variety of non-contact techniques for health monitoring. There are a few difficulties and confinements that are presently influencing the exhibitions of visual strategies with the camera sensor. Elements comprising climate impacts, such as downpour, light, thundering, and wind, and the ensuing vibrations should be investigated with regard to SHM. Camera position, number of cameras, blind spot, database complexity, etc., are a few limitations faced by camera-based SHM methods.

The DIC method was applied for the investigation of bridge health monitoring [122]. The performance strain gauge data and digital images were utilized for risk analysis of a critical bridge structure. It was reported that strain measurements are unaffected due to camera height. The pavement defects are also detected during this investigation. The implementation of a semantic texton forest combined with ML paradigms has been utilized to perceive the pavement damage using the visual data collected through parking cameras [97]. The information was gathered using two cameras: (a) an HP Elite Webcam selected to mimic a low-quality resolution for ending the camera and (b) a gray Blackfly 05S2M camera to fulfill current guidelines of the stopping cameras [97]. It was also observed that applied techniques require a large quantity of data for their training. However, rutting, defects, depressions, inclinations, etc., are also required to be integrated with the pavement health monitoring model, as discussed above. Because the stopping camera caught unintentional territories that hindered the speed of the observing strategy, Xu et al. introduced a procedure that recognized the correct region of interest (ROI) utilizing an inverse perspective mapping [92,123,124]. Park et al. [12] utilized VICON T-160 cameras to develop 3D displacement models. They showed that this system was quite useful where torsional and lateral displacements occurred simultaneously. However, the above methods had certain shortcomings such as a minimum of three cameras, range requirements, and reflective markers [12]. Oh et al. [113] also implemented a gesture capture system with frequency decomposition to record the dynamic response of the structure.

Several research works have been published containing the hybrid blend of vision sensors and image processing paradigms. An EOS 5D MKII camera system was used for measuring the displacement of cantilever beams. The line segment method and voting methods were applied to model deflection curves. Ye et al. [115] proposed multipoint pattern matching paradigms to detect the goals from the pictures recorded from the GE1050 camera. It was observed that vapor and illumination adversely affect vision-based systems. The full motion of the civil structure was tracked using vision-based methods [122]. Yang et al. [125] developed a paradigm for under-sampled data using Sony NXCAM at 240 fps. Fukuda et al. [119] developed a digital camera and processing software. Wu et al. [120] provided a framework for dynamic testing of civil structures through the target tracking system. Yoon et al. [121] proposed a consumer camera for a target-free, vision-based system for SHM and utilized three paradigms, viz., Kanade-Lucas-Tomasi, MLESAC, and eigne system. An autonomous deep learning model was applied for continuous health monitoring and to detect damages in steel bridges [92,123]. Images of bridge and element size were particularly investigated to understand the potential of the DL-based method [92,123]. Feng et al. [122] also evaluated multipoint displacement of building frame structures utilizing the unsampled correlation and OCM techniques. Luo et al. [97] suggested the requirement of the new application for the combined images and video analysis. InnVision technique was developed to handle dim light, multipoint displacement, and camera vibration problems. Xu et al. [92] utilized a low-grade camera to investigate the model frequencies of cable bridges. Xu and Brownjohn [100] reviewed the state of the art for SHM with regard to displacement estimation, tracking of targets, instrument calibration, and the associated challenges.

Chen et al. [125,126,127] utilized a high-speed camera for quantifying the mode shapes of civil structure through vibrational study. They applied an RGB-D camera for the detection of deformation in civil structures. RGB-D sensors were able to give their maximum performance at a 30 Hz sampling rate. In a separate study, a Kinect sensor was investigated to measure 3D translation motion along with torsional and rotational components for understanding the dynamic behavior. The noncontact methods provide potential application areas for modal identification. Feng and Feng [128] performed displacement measurements of civil structures through the stiffness and excitation forces. Kromanis and Al-Habaibeh [129] applied smartphones with an optical lens to continuous surveillance of vibrational movements in civil structures. Poozesh et al. [130] utilized optical data to implement a complexity pursuit algorithm for construct source signals. Molina et al. [131] applied the MM and DIC for the modal shape characterization of the stepped aluminum bar. The impact of weather conditions on camera performance was also studied using signal processing techniques, observed to have 1% variation as compared to accelerometers [132]. Zhou et al. [133] applied videogrammetry under variations in temperature and reported axial, horizontal, and vertical variations in temperatures and applied wavelet transform for the detection of signal frequencies. Yu and Pan [134] implemented the stereo DIC method on a single camera and reduced the cost by replacing two cameras with one. Yeum et al. [135] applied automatic picture collection through UAVs. They also proposed a new image localization method to extract ROIs. Yang et al. [136] developed a novel modal analysis algorithm where surface preparation was not required. Similarly, Oh et al. [137] utilized the Vicon 2016 camera to propose a new modal identification technique.

In a high-speed camera, noise may accumulate and affect modal analysis, acting as the main disadvantage. Noisy camera pictures were studied employing the least square complex frequency methods for modal analysis [138]. The integrated camera and accelerometer data provided excellent results for model parameter estimation. Javh et al. also applied spectral optical flow imaging for estimating displacement through a DLSR camera [139]. Khuc and Catbas [140] proposed a new workflow combining vehicle load as input for a beam-type or plate-like steel structure.

Feng and Feng [141] provided an extensive review on vision-based techniques describing system recognition, damage detection, its field application, associated errors, processing paradigms, etc. An integrated highspeed camera system was used to conduct a feasibility assessment for visualizing concealed damage. The concealed damage was subsequently visualized with a laser Doppler vibrometer scanning system [142]. Deep mining was employed to monitor SHM vision-based data. ML can deliver powerful scientific frameworks that will help simulate the performance and conditions of a civil structure. ML takes advantage of the potential to develop a revolutionary algorithm for structural health analysis and forecast. The above section reviews the application of cameras and video cameras for SHM with different processing algorithms. However, various problems directly hamper the performances of vision-based methods such as weather conditions, environmental vibrations, and difficulties in the detection of small-amplitude motion, which must be addressed properly. The chronological arrangement of important published works for camera-based methods is listed in

Table A4. The chronological arrangement of recently published works using camera-based methods.

Authors	Year	Camera Types	Numerical Strategy	Main Topics Researched	Reference
Chang et al.	2020	High-speed camera	Scattered wavefield reconstruction	Damage that has been hidden from view	[257]
Bao et al.	2020	Video camera	Image processing	Stay cable condition assessment	[234]
Bao et al.	2020	Digital camera	Machine learning	Symmetrical structure	[258]
Xu et al.	2018	Action camera	Encoding, software design	Stiffness of cable bridge	[92]
Javh et al.	2018	DSLR D850	ANN, CNN technique	Displacement measurement	[138]
Khuc et al.	2018	DSLR D5600	C and C++ Techniques	Deflection identification	[140]
Javh et al.	2018	High-speed camera	Software design	Nonlinear building frames	[139]
Xu et al.	2017	DSLR D7000	ANN technique	Crack detection	[100]
Chen et al.	2016	Digital video camera	Dynamic techniques	Earth retaining and tall structure	[126]
Santos	2016	Digital camera	Kalman filter	Stiffness of tall building	[47]
Yang et al.	2015	DSLR L18	Head segment interest	Present time crack detection	[93]
Oh et al.	2015	DSLR D5300.	Vision-based identification	Health assessment of building frames	[113]
Chen et al.	2015	Digital video camera	Motion magnification	Symmetrical linear structures	[127]
Trebuna et al.	2014	3D high-speed camera	Typical mode and complexmode marker work	Modular investigation	[259]
Im et al.	2013	Digital camera	GPS technology	Symmetrical structures	[196]
Olsen et al.	2010	Terrestrial laser scanner camera	Feature extraction by software	Damage assessment of the structure	[260]
Helfrick et al.	2009	Computer vision camera	Computerized picture connection	Shape strategy for crack recognition	[95]

(in Appendix A).

3.2. Smartphone-Based Electronic Packaging Approach

The AI chip undergoes super multi-parallel processing that mimics the brain. AI chips may need to secure massive I/O and electrode terminals due to high integration. As an ultra-fine bump forming technology, the electroplating method is the main method for forming solder. However, solder bump formation such as SnAg or SnCu via electroplating has difficulties in uniform bump composition and bump height, and plated composition is generally limited to binary alloys. Generally, Type 6 or 7 solder bumps offers excellent printing of the flip-chip interconnections in electronic packages, as shown in Figure 12.

The red arrows in Figure 12 show the position of the wafer where solder bumps are reflowed for the interconnections to the outer world in electronic packages [143]. For advanced, ultra-fine pitches, stencil printing is complex and associated with a number of variables that determine the yield and stability [143]. For high-density interconnections, due to the increase in functionality, accommodation of various chips together becomes difficult. It is expected that high-density wiring will be required for AI chips and line widths of less than 1 μm [144]. According to Lie et al. [145], packaging challenges in the wafer-scale for deep learning include cross-die connectivity; yield; mismatch in thermal expansion between the Si chip and PCB in the package; package assembly with precise alignment including PCB, connector, wafer, and cold plate; high power delivery; and cooling [145].

AI for mobile devices such as smartphones has been applied by Qualcomm, Apple, Huawei, MediaTek, etc., and AI for vehicles is applied by Tesla Co. with its hardware version 3 (HW3) for ADAS (advanced driver-assistance system). Tesla’s automatic pilot includes self-parking, driving lane centering, traffic-aware cruise control, and so on. An AI processor for next-generation is applied by IBM and Intel using neuromorphic chips [146]. Intel designed the Loihi chip to provide functional systems to implement SNN (spiking neural networks), which is a fifth-generation self-learning neuromorphic research test chip. The Loihi chip is known to include 130,000 neurons, each of which can communicate with thousands of others [147]. The Swiss company Sensimed introduced the Triggerfish^® contact lens sensor (CLS) to monitor continuous intraocular pressure changes for 24 hrs. The contact lens is made of soft silicone having a diameter of 14.1 mm and thickness in the center of 585 μm. Two strain gauges, a microprocessor, and an antenna are embedded in the lens [148]. The adhesive antenna sends patient information to a portable recorder. Hayashi et al. also reported CMOS, a self-powered, and fuel-cell-embedded continuous glucose-monitoring contact lens, where the footprint of the CMOS prototype is 0.36 mm² (600 μm × 600 μm) [149].

Modern cell phones are equipped with acceleration sensors, gyrators, and a global positioning system that can be skillfully applied for condition monitoring of civil structure buildings. There has been an expanding pattern of applying cellphone sensors in the health monitoring of structures because of their low price, versatility, enormous storage capacity, noteworthy computational capability, and effectively modifiable internal programming. Initially, efforts were made to recognize human movements utilizing Android-based cellphones. With the advancement of smartphones associated with a lot of attractive features, they are widely utilized for SHM of civil structures.

Wang et al. [150] utilized the iPhone’s camera, iOS application, and D-Viewer for capturing 3D displacement of a building structure. A camera was employed to continuously monitor a spherical target, and the iOS application estimates objects’ directional displacements. Zeng et al. [151] developed an application for smartphones to predict road roughness for the transportation department of the US state of Virginia. Zhao et al. [152,153,154] reported the use of smartphones for quick SHM of a bridge. They proposed vision-cable force measurement techniques that were implemented using the iPhone camera and authenticated through the cable model test. D-Viewer and Orion-CC are two iPhone operating system applications that were developed especially for SHM and freely available.

The fourth generation iPhones were used as a mini-SHM system containing embedded responding software and inter and outer sensor board configurations [155]. The internal sensors collected information about temperature, inclination, moisture, azimuthal, and acceleration which were further processed to make rational decisions regarding objects under surveillance for SHM. If inter-board configuration fails to fulfill SHM requirements, then outer sensor board configuration was applied with port or Wi-Fi connections. To validate the iPhone’s Gyroscope, a swing test was also conducted with dynamic angle and inclination measurements [156]. A shake table test was executed to assess the reliability of the iPhone as seismic monitoring equipment with 1D and 3D tremors ranged between 1 to 10 Hz, relocating earthquake scenarios. They presented a novel application of smartphones for measuring intensity parameters of ground motion utilizing four 3GS iPhones and three iPod touchpads [157]. The primary drawback observed for the iPhone in earthquake monitoring was its limited operational range.

Yu et al. [158] proposed SCHS stereo-DIC methodology with four mirror adapters for the measurement of 3D dynamic measurement. Surface images of test and target objects were measured through two dissimilar ocular pathways and later processed to obtain the vibrational response of the object or specimen surface. The vibrational factors such as damping ratios, mode shapes, and natural frequencies were also estimated to validate the potential of the proposed approach. It was found that the proposed approach was practically effective for dynamic parameters estimations and vibrational measurements. A new cable force estimation technique employing an iPhone with an installed Orion-CC application was proposed for SHM [134,159]. A comparison study was also conducted between iPhone and wireless monitoring methods then Orion-CC was implemented on lab cable model tests. Finally, the proposed technique was implemented for the health monitoring of the Dalian Xinghai Bay Cross-sea Bridge to prove its efficacy in pragmatic situations [160].

A mini-SHM system was developed with embedded responding software and inner sensor board or exterior sensor board configurations on iPhone 4S which was connected with sequential port or Wi-Fi. Cable force and swing tests were conducted in the laboratory and the cable test was repeated on the real bridge to validate the applied method. Various examinations were led on a research center level utilizing a shake table and real field extensions to confirm the practicality of the suggested sensor board. Innovative mobile testing methods were applied for SHM using iPhone 4S with two new data acquisition methods. Later, a cable force test was performed to validate the anticipated method. Finally, the developed method was also implemented on the Hualu and Sifangtai bridges to authenticate its efficiency for fast cable force measurements.

Zhao et al. [152,153,154] investigated structural displacement using the laser projection sensing technique. They processed structural displacement data with D-Viewer developed on the Android platform. They compared two typical cell phones, namely, Samsung A5 and Meizu MX4. They were utilized for the measurements of structural displacement in static and dynamic conditions. Samsung A5 demonstrated a higher securing outline rate because of its fewer pixels as compared to Meizu MX4. The trial outcomes demonstrated a blunder of 0.85% in uprooting estimation, whereas for the engineered overpass model, the error was 6.33%.

Zhou et al. [29] review thermal load and its distribution in bridges with emphasis on numerical analysis and field estimations. The heat transfer in the bridge with boundary conditions was first discussed with finite difference and finite element methods. This study considered steel, concrete, and steel–concrete bridges for thermal load analysis. Smartphone applications were reviewed for economical unpleasantness observing of street surfaces [155]. Twin Android-based tablets were utilized for collecting information. They were firmly retained in the crates fixed on the vehicle floor during the trial trips. The utilization of cell phones requires particular preparation in programming and PC programming alongside broad stockpiling instrumentation to process large information productively [155]. An Android system APP was developed that allows many Android cellphones to be quickly converted into a wireless SHM system [155]. The server/client architecture was used to make the planned system stable and simple to use. A smartphone was designated as the system’s server to control all other smartphones, which served as structural vibration sensors. A comprehensive review of vision-based SHM concepts, techniques, and their real-world applications has been published [161].

The critical findings of smartphone strategies rely on the following: (1) it has an effective programmable programming stage, (2) continuous observation is conceivable over the web and distributed storage, (3) advanced cell direction is basic for worldwide evaluation, (4) it is reasonable and alluring for huge information assortment, and (5) it can work with no preparation. Recently, CNNs have been successfully applied as efficient ways for feature extraction, advancing image categorization, and object identification technologies [162]. Wang et al. proposed a method for detecting beam fractures based on acceleration waveforms using a DL model. This model was trained using 20,000 and 200 pseudo acceleration waveform datasets generated through simulation, respectively [162].

Table A5. Recent publications on smartphone-based methods applied for SHM purposes.

Authors	Year	Phone Types	Numerical Strategy	Researched Topics	References
Wang et al.	2021	Android	CNNs	Steel beam analysis	[162]
Dong et al.	2020	Android	Computer vision-based	Local and global stiffness analysis	[115]
Zhang et al.	2020	Android	Vibration monitoring	Seismic structure analysis	[155]
Wang et al.	2018	Apple 6s	Stiffness detection	High-speed camera	[150]
Khuc and Catbas	2018	Apple 5s	Damage identification	Health monitoring of bridge	[140]
Yu et al.	2017	Apple 4S	Crack detection	Seismic wave measurement	[134]
Zeng et al.	2017	Tablets with Android	Android utilization	Pavement cracks detection	[151]
Feng et al.	2017	Apple 4S	Hybrid motion sensing	High rise structure	[128]
Zhao et al.	2016	A5 Android	Android utilization	Stiffness detection by laser technique	[153]
Ozer and Feng	2016	Android Apple 4S	Displacement measurement	Assessment of bridge health monitoring	[261]
Matarazzo and Pakzad	2016	Apple 4s	Crack identification	For multi-story frame structure	[262]
Marulanda et al.	2016	Android	Ambient excitation	For seismic structure	[263]
Dashti et al.	2014	Apple 3G	Earthquakes checking	Seismic measurement	[264]
Höpfner et al.	2013	Android	Smartphone sensors	Measuring mechanical oscillation	[265]
Tang et al.	2002	Android	Mobile manipulator	Bridge crack inspection	[266]

shows recent publications on smartphone-based methods for structural health monitoring (in Appendix A).

3.3. Unmanned Aerial Vehicle Method

A large quantity of contact sensors is generally required for SHM of civil structures, which limits the affordability of contact sensors for large area coverages. Primarily, UAVs or drones are utilized in the field as an affordable alternative to contact sensors. UAV-based measurements have gained support from researchers due to advancements in control strategies, robotics, computational power, and real-time automation capability. Light-weight cameras are mounted on drones to capture image and video data to evaluate the structural health, and contain vision-based systems consisting of GPS, optical sensors, infrared cameras, LIDAR, and navigation systems. They also have airborne flight data capturing and processing capabilities that can be easily controlled from the ground. UAV applications reduce data acquisition efforts, conventional logistics, and accidents. They provide better spatial and temporal resolution of images when compared to satellite images. Three-dimensional images of civil structures can also be captured by mounting terrestrial laser scanners on UAVs, which is an advantage for larger structures.

UAVs were utilized for traffic surveillance and bridge inspection with a new control law for tacking the targets [163]. Rathinam et al. [164] applied image sensors to the drone containing a tracking system and GPS controls. The tracking paradigm did not consider the weather impacts such as wind, rain, etc. A UAV installed with a digital imaging system was established to gather road surface data to create a 3D model to assess interior damages [165]. During an experimental flight, a multi-rotor UAV captured photographs of various rural roads with symptoms of worsening such as pits, potholes, and furrows. The 3D coordinates of conjugate points were computed through two stereo pictures employing image processing techniques, including image orientation.

Roca et al. [156] inspected building facades and roofs with UAVs fitted with Kinect cameras. The photos were heavily overlapping, the same scene obtained from multi-view matching (around 90–95%), to create a point cloud picture of the façade (3 to 4 m of flight height) [166]. A helicopter furnished with a camera was used to measure the UAV’s use in monitoring unpaved road surfaces [157]. The UAV could fly between predefined waypoints automatically and maintain constant flight in winds up to 5 mph. The photographs were taken on a 200 m length of road in 5 min at 2 m per second [157]. Later, the pictures were processed through Canny and Hough circle algorithms to identify potholes and their radii. Eschmann et al. [167] applied non-destructive testing (NDT) for civil structures using UAVs, specifically micro aerial vehicles (MAVs). An octocopter attached camera with 12-megapixel resolution was chosen for the building assessment. Instead of employing GPS navigation, the MAV was operated manually. An automatic image capturing sequence was used to capture photographs of the building. There were two methods, namely, edge detection and Gaussian Blur, used to study automated crack identification; however, these were insufficient in sensing smaller cracks or fractures. It was also studied how UAVs moved horizontally and vertically in the direction of data accuracy to see whether UAV-based SHM is feasible. Ortiz et al. [168] deployed UAVs to monitor the health of cultural sites. The UAV captured video, conducted a thermographic study, and took temperature readings. Several weathering patterns were discovered by the UAV throughout the flight, including material loss, fissures, color loss, unwanted scale deposits, corrosion, erosion, etc. Ellenberg et al. [169] employed 3D simultaneous mapping and localization technologies through UAVs. They used the UAV to validate field demonstration for crack recognition in an ordinary bridge. A new method termed oblique color imagery was devised to record the cracks or fractures on the building fronts, which were ignored in prior publications. Galarreta et al. [170] utilized UAVs to capture high-resolution oblique photographs as a remote sensor. The photographs were utilized to create a 3D point cloud for damage assessment and to examine the roof and facade using object-based image analysis. The key issue was the integration of damage datasets acquired from multiple portions of the building structure. An examination of the 3D point cloud with proper picture diagnosis at the same time could be a feasible answer, but more research is needed. In another study, a UAV was tested on an office building’s frontage and navigated the confined areas and delivered optical recognition of fractures with a short exposure period to decrease motion fuzziness [171]. The fissures, which were 0.5 mm wide, were visible. Sankarasrinivasan et al. [172] applied a grayscale filter based on hat transform methods to locate cracks. To offer effective crack identification, a Hat transform was combined with a hue filter, e.g., hue saturation value, having an appropriate threshold. Wind and picture noise were handled using modern flight controllers and gyrostabilized cameras. Zhou et al. [173] applied a graph cut algorithm for 2D image processing to detect road pit holes. The tests were carried out with 2760 images at diverse resolutions and flight ways. The precision of the UAV system was assessed to be 98.4%, indicating that UAVs have large future potential. However, drift problems and uneven contours were proven to cause large errors at low speeds in UAVs.

Cho et al. [174] executed crack recognition utilizing CornerHarris, a feature-based image recognition technology that uses Haar features and subsequently converts the pictures from color to grayscale. On black and white images, histogram equalization was employed to improve recognition rates along with an adaptive binary approach which creates a threshold on the input image data. This method safely examined skyscraper buildings and can also be used to scan cliffs and docked vessels in other industries. Researchers looked at post-data collection techniques such as data processing, vision-based approaches, and geometrical processing to find the most important photographs [175].

The UAVs were equipped with a vibrational non-destructive technique (NDT) for the early detection of cracks to lower both maintenance expenses and the sensors installed on the civil structure [176]. The planned NDT approach relied on piezoelectric material that assisted as both an exciter and a sensor. It was wired to the UAV and magnetically connected to ferromagnetic materials for wood or concrete civil constructions. The developed method was also successful in distinguishing between distinct damage types. Franke et al. [177] investigated two dissimilar soil liquefaction locations in Chile using small UAVs.

Qidwai and Akbar [178] detected flaws in metallic constructions through UAV inspection which was paired with a robotic magnetic flux leaking device. Edge detection and Hough transform were used to create the imagery of metallic construction. Reagan et al. [179] identified cracks in a concrete bridge using UAVs and 3D DIC technology. The 3D DIC UAV system was shown to be a viable SHM solution. Reagan et al. [180] investigated the UAVs and 3D DIC techniques for the SHM of a bridge. Laboratory and field investigations were used to verify the suggested system’s evaluation. It was observed that proper lighting conditions are essential to capture the data accurately to build a 3D DIC system.

The utilization of unmanned aerial vehicles (UAVs) for large-scale structure monitoring was investigated [181]. The researchers employed a remotely operated airborne vehicle to detect the displacement of a huge floating membrane in a wastewater treatment plant that spans over 170 × 420 m². The UAV system only gave structural displacement concerning the camera’s position. Yoon and Spencer [182] combined relative structural displacement with camera motion, and a framework was developed to obtain the structure’s absolute displacement. Damage detection in inaccessible places is the main area for UAV applications. Morgenthal and Hallermann [183] used UAVs to detect damage, which was an improvement above the traditional inspection method. A brick construction, a hangar, a turbine, and a chimney, displaying diverse damage types, were all subjected to test flights.

Ellenberg et al. [184] successfully merged the imagery received from UAVs. Researchers were able to conduct a bridge study and achieved quantitative facts about civil constructions. Kim et al. [185] presented a fracture recognition technology founded on UAV-acquired pictures and image processing. The data of concrete walls containing diverse crack types, caused by loading creep and shrinkage conditions, were acquired from the field. The photos were then processed using a hybrid image binarization technique to determine the crack width. With an inaccuracy of 7.3%, the planned picture processing approach was effective in defining cracks with a thickness greater than 0.1 mm. Omar and Nehdi [186] utilized UAVs furnished through infrared thermography abilities to observe defects of strong framework surfaces. The captured warm pictures were prepared using an estimation that sewed the photos composed to outline a variety of the augmentation surface, and a k-means bundling strategy was applied to order the faults into essential groups. Figure 13 demonstrates the components of the UAV method for structural data collection.

UAVs outfitted using infrared thermography capacities were utilized to observe the defects of solid extension surfaces. The captured pictures were prepared to apply a calculation that sewed the pictures composed to shape a variety of the scaffold surface and the k-means bunching method was utilized to order the deformities into serious gatherings. Germanese et al. [187] proposed a health monitoring of the structure system for striking structures using UAV captured pictures for crack detection. The test game plan was driven in an examination office by utilizing indicators to detect crack openings. These indicators were relocated all through the examination to impersonate time-consuming crack courses of action.

In [118], damage identification was developed using a diminished scale channel called the top change method. The authors tried to merge a base top change with a color-based channel, for instance. The UAV sensors are sensible for observing hard-to-arrive-at zones and give remarkable transient and spatial objectives conversely with satellite pictures. One of the appealing features of UAV devices is that they give 3D data that are important for tall structures. A visual vibration checking framework was introduced to screen the speed and relocation area [117,118]. The picture preparation was carried out using three unique strategies: outline contrast technique, molecule picture velocimetry, and optical stream strategy [118]. The test after effect of the investigation indicated that the optical stream technique arrived at higher precision contrasted with molecule picture velocimetry. This technology applied a sliding window-based relationship calculation that was exceptionally needed due to the characteristics of captured pictures.

In a comparative report [188], virtual visual sensors were applied through digital cameras for health monitoring of timber wooded structures. Vibrational frequencies were estimated by the intensity of the fixed pixel coordinate of captured video and then by implementing fast Fourier transform to extract natural signal frequencies. Degradations in stiffness and weight of wooden material are reflected in the natural frequencies. Visual sensors were applied to monitor the health of a United States Forest Service’s pedestrian bridge. It was found that moisture content and damage on the bridge had a significant impact on the natural frequencies. In an additional investigation [173], street recognition was performed through UAV by applying a graph cut algorithm on UAV aerial videos. The investigations were performed on cleared streets with 2760 pictures being caught in changing goals and strategies for flight. The street detection framework provided 98.4% accuracy with a processing speed of 43 frames per second with an average of 1046*595 videos. Morgenthal and Hallermann [189] achieved damage recognition utilizing UAVs to enhance customary investigation techniques. Practice runs were completed on a stonework fabrication, a holder, a breeze turbine, and a smokestack, all displaying different types of damage. They discussed visual inspection of civil structures using UAVs with the effects of wind speed on the image data. They applied computer vision procedures for the detection of cracks in various civil structures such as a church, chimney, under bridge unit, truck crane, etc. They completed an examination on connection and obtained quantitative subtleties of common structure frameworks by joining symbolism gained from the UAVs.

Kim et al. [185] also offered a crack recognition workflow using a UAV captured video data with a hybrid image processing technique. The tested UAV contained an ultrasonic sensor, a camera, and a small Wi-Fi unit for capturing images of the target structures. This proposed approach detected fractures thicker than 0.01 mm with a prediction error of 7.3% for length [190]. Duque et al. [191] checked lumber connects utilizing a UAV joined with a fracture evaluation convention. In [192], the crack central point technique was proposed for the fracture detection of a bridge structure using UAV-captured image data. They also reported comparative results of the proposed technique with the k-means clustering technique and the edge detection method to establish its supremacy as more robust and adaptable technique.

In [192], machine learning approaches were utilized to improve the different design and functional elements of UAV-based communications, such as network modeling, source management, placement, and safety. In [193], ML, stochastic geometry, optimization theory, transport theory, and game theory are present among the analytical frameworks and mathematical techniques discussed. The usage of such technologies to solve specific UAV-related challenges is also discussed. In a nutshell, this research work explains the optimization, analysis, and development of wireless communication networks based on UAVs. The picture recording and geo-tagging system along with CNNs were used to quickly identify and locate damage [194]. The geotagging of 3D coordinates and camera posture data with bridge examination photographs was possible due to the image capturing and geotagging. DL-based CNNs were trained for automated crack detection. In [195], a path planning method for multiple UAVs to cooperatively track targets using a DQN-based MADDPG (Multi-Agent Deep Deterministic Policy Gradient) paradigm, which can dynamically plan and adjust the flight path of multiple cooperative UAVs in real time and achieve better tracking effect over time. Im et al. [196] published a comprehensive review of GPS technology for SHM applications.

Table A6. Recently published paper for UAV applications in SHM.

Authors	Year	Type	Methods	Main Topics Researched	References
Liu et al.	2021	Drone	ML	Confrontation of multi-UAV assault and defensive	[195]
Saleem et al.	2020	UAV	CNN	Bridge visual inspection	[194]
Mozaffari et al.	2019	UAV	ML	Channel modeling, resource management, positioning, and security	[193]
Germanese et al.	2018	Drone	Compact system camera	Damage design in an old building	[187]
Duque et al.	2018	UAV	Digital image processing	Detection for bridge deterioration	[191]
Lei et al.	2018	UAV	A digital camera with acoustic sensors	Crack detection and bridge inspection	[267]
Omar et al.	2017	Inspire T600	Infrared imaging camera	Detecting of extension deck	[186]
Chiu et al.	2017	UAV	Digital camera	Tall structure monitoring	[181]
Qidwai et al.	2017	N/A	Quadcopters	Health monitoring of civil infrastructure	[178]
Radopoulou and Brilakis	2016	Drone	Quadcopters	Detection of multiple pavements	[102,103]
Na and Baek	2016	Camera drone	NDT testing	Large civil infrastructure	[176]
Ellenberg	2015	Camera drone	Action camera	Crack measurement	[169]
Sankarasrinivasan et al.	2015	N/A	Quadcopters	Damage measurement in a tall structure	[172]
Hallermann et al.	2015	Ascending Technologies	Mirrorless camera	Assessing spans	[171]
Franke et al.	2014	Drone	Small aerial vehicle	Geotechnical site investigation	[177]
Dobson et al.	2013	UAV	Quadcopters	Unpaved road evaluation	[157]
Roca et al.	2013	Skyjack	Small aerial vehicle	Outdoor assessment of building frames	[156]
Ortiz et al.	2012	Drone	Quadcopters	Civil structure surveillance	[168]
Rathinam et al.	2008	N/A	Aerial vehicle	Linear structure assessment	[164]
Metni and Hamel	2007	N/A	Small aerial vehicle	Bridge damage assessment	[163]

displays the rundown of the UAV framework and its significant implementations in the SHM domain in literature (in Appendix A).

3.4. Internet of Thing (IoT) for SHM

Continuous health monitoring of infrastructures is performed to maintain their structural integrity, sustainability, and serviceability for a long period of time. The sustainability of civil infrastructure has become an important matter worldwide because of the large quantities of civil structures that are required to be maintained and the efficacy of conventional maintenance and repair procedures approaches is questionable. This creates a strong platform for the advancement of newer technologies such as the IoT [197,198,199,200]. Figure 14 displays the architecture of the typical IoT system, having four key gears: (i) WSNs, (ii) Gateway, (iii) Remote Control and Service Room (RCSR), and (iv) Open Platform Communications (OPC) client server [201,202].

In IoT, WSNs are utilized successfully for structural monitoring, where the captured data are processed via smart software and local computer power [198]. IoT intends to expand machine-to-machine communication using WSNs to monitor and control devices remotely and efficiently. Smart distributed network of gadgets integrates and communicate inside its architecture to analyze information through cloud computing platforms. This also utilizes sophisticated software to extract relevant information from a wide range of data. ML can also be associated with IoT networks for detection, recognition, and localization of damage present in civil infrastructures. The application of the ML and IoT combination has become an important tool for handling SHM-related problems [198,199,200]. The key challenges for the SHM of civil infrastructures are the continuous monitoring of the sensors deployed and the comparison of new data with earlier readings [193]. The geographical separation also increases the difficulty for SHM. Therefore, a recording gadget is required that connects all sensors on the civil structure. In addition, the links between the captured information and a central monitoring station that can receive data from the sensors through the Internet are crucial. It is possible to successfully handle the stated issues using the combination of IoT and artificial intelligence technologies [200]. IoT allows engineers to gather data for future analysis from many bridges. For analyzing and interpreting captured data from WSNs, ML can be utilized. IoT-based structural health inspections may provide a promising solution for SHM systems that are fast, accurate, and low cost [201]. The integration of SHM, IoT, and cloud-based computing can lead to sophisticated data diagnosis. Cloud platforms can act as storage, and use intelligent monitoring devices on an SHM system. The real-time health status of the civil structures is communicated to an Internet server. The captured and saved data on the server can be viewed and interpreted using ML remotely from a mobile device [202]. The installed components are described as “things” that are identifiable, communicable, and interactable within the network. These smart IoT “things” have communication services with physical features, a unique identifier, IP address, and elementary computing capabilities, and they record physical phenomena, trigger actions, or actuate control [201].

4. Recent Concept of Smart Cities

In recent years, in many engineering groups, the notion of smart cities has become more important, and research is being developed via the implementation of the IoT concept to smart cities. The primary objective of a smart city is to utilize public resources efficiently and minimize operative expenditures and resource wastages. An intelligent city aims to make infrastructure smarter to optimally utilize its available resources. An IoT system installed in a smart city may offer distributed information to evaluate the structural integrity of monitored infrastructures using captured real-time sensory data. Further, data interpretation can be performed through DL architectures, such as CNNs [203,204]. The real-time data collected vary in structure type and value, and it is impossible for a particular system to analyze all these data efficiently for any city. It is important to use intelligent data interpretation approaches since every city is unique and has various associated issues and challenges [203]. It is important to provide strong data-collecting layers, communication protocols, data storage, etc., to handle SHM data. These vast volumes of SHM data can be processed through the DL approach to extract useful information. DL can be deployed for training computers to identify fault patterns of huge real-time WSN-based networks to provide early performance concerns [204]. The main difficulty faced by the smart cities’ idea is to cope with the huge volume of sequential data, time-series data, etc., captured through connected sensors. DL architectures are capable of processing sequential data [204]. DL platforms can also handle optimization challenges affecting smart cities and their building structures [205].

The concept of smart cities is based on their real-time monitoring and efficient management through the information gathered from a distributed network of sensors through various electronics packages as shown in Figure 15. The collected data will be processed through the AI platform for quick decision making that will help to ensure the sustainability, security, and efficiency of civil infrastructures [206]. Technological advancements have resulted in the creation of self-sensing materials that can offer cities more reliable and smart ways for capturing monitoring data to identify the current status of their infrastructures [206]. Another promising development is smart concrete, which may introduce self-sensing abilities or features in the civil infrastructures [207]. The correlation between the fluctuation of the inner stress and changes of suitable material characteristics helps to achieve such useful qualities. These sensors are made of a cementitious matrix with carbon nanotubes that can be employed for smart sensing purposes [208]. The interpretation of captured real-time monitoring data can be executed efficiently with intelligent algorithms [209,210,211]. The vision-based DL paradigm can be utilized for corrosion detection in concrete buildings.

The objective is to monitor the concrete condition during the curing time, resulting in enhanced life and safety of concrete buildings [192]. These sensors are implanted inside the concrete to capture the infrastructure responses which are transmitted to smartphones utilizing IoT. The collected data can be processed through AI algorithms for structural health monitoring.

5. Current Challenges and Future Perspective of SHM Domain

Civil infrastructures normally have high exposure to weathering effects and fluctuating loading situations that introduce fault, cracks, seepage, etc. [212,213,214,215,216,217]. SHM is performed for repairing and strengthening the cracks, faults, etc., for overall structural damage control [218,219]. These methods involve the application of sensors to monitor the current conditions, dynamic reactions, damage locations, and the evaluation of structural health all over the lifespan [6,7].

Data acquisition for structural information is also a risky task, as most of the monitoring equipment is installed on tall civil structures. Moreover, the captured information needs evaluation because of the multifaceted complex nature of information broadcast or interval management or control utilization [8,9]. Exceptionally precise sensors, optical and remote systems, GPS, and different advancements have all added to the improvement of more exact and cost-efficient checking of structures [12,13]. As a result, the volume of informational data has expanded immensely, at the pace of a great many estimations for each sensor.

The SHM data captured through modern sensors are known as “big data” and create challenges for existing data processing technologies. However, advanced sensors-based methods have their limitations and shortcomings. In camera-based SHM, there are a few difficulties and confinements that are at present influencing the exhibitions of visual strategies with the camera sensor. Elements comprising climate impacts, for example, such as downpour, light, thundering, and wind, and the ensuing vibrations should be investigated with regard to SHM. Camera position, number of cameras, blind spot, database complexity, etc., are a few additional limitations faced by camera-based SHM methods. To overcome the issues of camera-based methods, modern smart cell phones are furnished with acceleration sensors, gyrators, and GNSS receivers that are proficiently applied for the conditional appraisal of civil structures.

The basic challenges—namely, restricted lighting, multipoint relocation, and camera vibration related to vision sensors—limit the performance of smartphone applications [220,221,222]. However, certain limitations are also reported for UAV-based SHM methods such as difficulties in finding an appropriate UAV position, environment conditions, the stability of the UAV, design challenges, motor faults and failure, limited control range, etc. Moreover, discrimination of acoustic signals from background environment noise is a difficult task for acoustic sensors due to their weak strength. These sensors are embedded types that cannot be utilized for other purposes after installation. Reliability is also a major concern for acoustic emission sensors. To handle the complex sensory data acquired for SHM from civil structures, artificially intelligent models are recommended for the processing and extraction of significant information from these captured datasets. The performance of AI models depends upon the superiority of data acquired from the sensors. The faulty sensors introduce noise which is an undesirable and serious issue for ML models such as noise adversely affects their performance. Several research works contain diverse ML techniques for the detection and localization of faults; however, none of them have comprehensively compared the different ML techniques on the same data types. Figure 16 shows a typical SHM systems along with its components and techniques involved.

Given the aforementioned obstacles, it is quite difficult to judge the effectiveness of intelligent techniques such as ML available in the literature [223,224,225]. Secondly, the selection of suitable values for the parameters of machine learning models is a crucial task for robust application in SHM. For example, to maintain their impressive performances, fuzzy logic requires its member functions to be well defined, ANN depends on the right selection of its architecture, SVM needs its hyperplane parameters to be properly tuned, etc. Here, chances of overfitting are also quite possible as bulk data are available for the training phase. There are no proper guidelines available for the choice of a suitable AI model for a particular application in SHM. ML models are found unsuitable for vision-based systems.

With the advancement of DL, smart sensors, and smart monitoring systems, several shortcomings have been minimized and research is ongoing for their further development. Smart sensors have enhanced the quality of measured raw data and reduced their noise content. More research is required to develop better smart sensors as they still require calibration from external processors. DL models are capable of handling big data with inbuilt feature extraction characteristics that reduce computational costs. They do not require separate parameter tuning in general, which is essential in ML models. DL models, e.g., CNNs, are highly suitable for machine vision-based applications such object detection, facial recognition, etc. Furthermore, with the emerging concept of reinforcement learning, transfer learning, edge computing, and cloud computing, new technologies will evolve in the SHM domain.

6. Conclusions

Predictive maintenance operations are performed using advanced data-driven SHM systems that utilize real-time and up-to-date information on civil infrastructure conditions. This, in turn, reduces the overall maintenance costs as interventions would only be performed when required. Hence, it has become possible to attain smart and more sustainable civil infrastructure by minimizing the resources and funds required for its regular maintenance. We carried out a comprehensive review on data acquisition methods and AI models applied in SHM to maintain sustainable civil structures. Data acquisition methods are reviewed that made the utilization of intelligent paradigms easier during SHM. AI models are evolving in parallel with the advancement of smart sensors, and both are highly interconnected with each other [226,227,228,229,230]. More accurate data can be captured with modern sensors that require sophisticated AI algorithms as a processing tool. Therefore, the utilization of next-generation sensors—e.g., high-resolution cameras, drones, automated sensors, cell phones, etc.—has been reviewed explicitly for health monitoring of the civil structures. This study reports the pros and cons of diverse data acquisition techniques and AI models along with constraints associated with SHM. Several critical findings for the prevailing sensor-based data acquisition techniques can be summarized as given below.

• Contact sensors are often utilized for SHM due to their smaller size, lower economic cost, and less power consumption. However, the discrimination of signals from background environment noise is a difficult task due to its weak strength. These contact sensors are embedded types that cannot be utilized for other purposes after installation. Reliability is also a major concern if any sensor becomes faulty.
• Several non-contact sensors are preferred over contact sensors for SHM due to their technological advantages. Camera position, the number of cameras, blind spots, database complexity, weather conditions, etc., are few limitations faced by camera-based SHM methods.
• Smartphones are low-cost cell phones that are utilized for SHM due to their versatility and enormous stockpiling limits, noteworthy computational force, and effectively modifiable programming. However, they require particular preparation in programming and PC programming alongside broad stockpiling instrumentation to process large volumes of information productively.
• UAVs equipped with a lightweight camera are primarily used for imaging and, consequently, measuring an area with the overall prosperity of a structure. Certain limitations are also reported for UAV-based SHM methods, such as difficulties in finding an appropriate UAV position, environment conditions, the stability of the UAV, design challenges, motor faults and failure, limited control range, etc.
• Other non-contact sensors, such as laser vibrometers, LIDAR, and radar interferometry, have been reported to give excellent estimation results for SHM. These instruments are reported to be costly and adversely affected by rough climate conditions.
• Machine learning models are found unsuitable for vision-based systems. However, deep learning models have been widely applied to vision-based SHM systems, allowing them to deal with large real-time datasets.
• Unmanned aerial vehicle and non-contact sensors are found to be the most promising smart data acquisition technology, whereas convolution neural networks comprise the most impressive data-driven models reported for SHM.
• With the progress of IoT integrated with AI algorithms, health monitoring of civil structures has become a much easier task compared to traditional SHM systems. Algorithms such as CNNs are continuously monitoring civil structural integrity and may schedule maintenance to minimize any damage in the structure.
• IoT-based structural health inspections may provide a promising solution for SHM systems that are fast, accurate, and low cost. The integration of SHM, IoT, and cloud-based computing can lead to sophisticated data diagnosis. Cloud platforms can act as storage, and use intelligent monitoring devices on an SHM system.

We studied several AI applications in SHM to deliver a broad review of the existing technologies and advancements. ML models are mainly utilized to extract useful information from bulk health monitoring data. Supervised and unsupervised ML models have been reported to be utilized for health monitoring of bridges, buildings, pipelines, etc., to maintain their structural integrity. With the advancement of technology, more complex AI models have been applied, from simple K-NN to sophisticated CNN models for SHM. Convolution neural networks comprise the most advanced and impressive data-driven models applied for SHM. Overall, AI models have given impressive results for the handling of bulk structural monitoring data. However, more comparative studies are required to test various intelligent models on SHM datasets to demonstrate their effectiveness.

These models also offer various advantages, such as integration with the Internet of Things. Moreover, ML can be applied for solving optimization problems. These models are computationally efficient and require limited data samples during their training phases. Finally, most of the ML models applied for SHM have produced impressive simulation results; however, only a few of them are tested in real-time dynamic conditions for real-world systems.