A Systematic Review of Sensing Technology in Human-Building Interaction Research

Abstract

Human-building interaction is an emerging field of study that investigates the interactions and reciprocal impacts of humans and building systems. In this discipline, sensing technology is critical for data collection. The application of sensing technology is divided into six categories based on the research topics: (1) occupancy status, (2) occupant physiological indicators, (3) building components, (4) building environment, (5) building consumption, and (6) fusion of multi-sensing system. By evaluating 127 relevant research articles, this study attempts to provide a systematic review of the implementation of sensing technologies in each HBI research topic. Four significant sensing technologies were investigated for the occupancy status study: camera-based sensing, infrared-based sensing, radial frequency signal-based sensing, and ultrasonic sensor. Methodologies for biosensing brain activity, muscle and skin function, and cardiac function were examined as occupant physiological indicator measurements. The magnetic reed and vibration sensors were discussed for sensing changes in building components. The air property sensor, sound sensor, and illuminance sensor were introduced to monitor the building environment. The smart meter and smart plug were examined for sensing building consumption, and the application of multi-sensor fusion was also included in this article. Furthermore, this systematic study discussed three aspects of contemporary sensing technology deployment: data concealment, sensor cost tradeoffs, and privacy concerns.

1. Introduction

Human building interaction (HBI) is an emerging interdisciplinary research area that has attracted the attention of scholars from both the field of computer science (CS) and building. From the view of CS, HBI derives from human–computer interaction (HCI) research and focuses on the built environment [1]. Triggered by various electrical and information technologies, buildings are becoming more sensible, responsive, and smarter, just like a giant computer in the shape of a building; therefore, scholars from CS are trying to examine the involvement of HCI in the building study, and HBI will focus on human’s values, needs, and goals when addressing human’s interactions with the smart built environment [2].

The research of HBI has definitely attracted the interest of scholars from the building field. Compared to the CS group, the building group is concerned about, but not limited to, the smart building. They care about all kinds of interactions between humans and the building system, called occupant behaviors, such as opening windows, adjusting the thermostat, turning on lighting, etc. Sharing with the similar concept of HCI research, in HBI research, occupants are placed in the center of building design and operation, meaning that occupant’s behaviors, perceptions, feedback, and experience are considered in advance and integrated into the design scheme and the building control system. Aiming to figure out the hidden psychological and social factors in the occupant’s interaction with the building, a number of interdisciplinary studies with psychology and social science have been carried out and have given rise to a new research paradigm, referring to the research frameworks proposed by Hong et al. [3] and D’Oca et al. [4].

According to the summary of Norouziasl et al. [5], the current HBI research comprises two aspects: (1) the occupancy, which involves the counting, the trajectory, and the presence of the occupant; and (2) the occupant behavior to control or to adjust the components of building system, including the window, door, lighting, HVAC system, etc., which is denoted by occupant interaction. One important reason for the building community to conduct HBI research is to improve the performance of building operations. Building sectors account for 40% of the total energy consumption in the world and plenty of studies have been carried out to investigate the energy issues of buildings [6]. However, the actual energy savings of a building cannot be guaranteed by the energy efficiency technology or policy alone, and occupant behaviors are the crucial points deciding the actual energy consumption and indoor comfort [7]. Plenty of existing studies have contributed a lot to strengthening the understanding of occupant behaviors in the building, including the behavior modeling method, the energy simulation integrated with behavior, the correlative factors of occupant behaviors, and the behavioral intervention for energy saving [8,9].

Occupant behavior is an important aspect of HBI research, the trigger of the behaviors is affected by a series of factors involved with the building system and environmental conditions. HBI research is now facing a more comprehensive system, integrated by three subsystems—human, building system, and environment (HBE), which are both independent and interrelated with each other. In addition, HBI research investigates the running status of the whole system and explores the interactive relationship between each subsystem. It is crucial to gather information covering the overall human-building-environment system for researchers to reveal the running patterns of both the whole system and subsystem, as well as the mutual linkages between each subsystem. The rapid development of sensing technology has provided scholars with new opportunities to collect more comprehensive information on the whole system of the human-building-environment, such as the vibration of the building structure, the status of the windows, the indoor air quality, and the building energy consumption, etc. Additionally, except for the occupant actions or occupancy, the physical sensations, the psychological feelings, and the user experience of occupants also affect the interactions between humans and buildings, further influencing the building’s operation, which is worthy of deep investigation. Traditionally, a self-reporting questionnaire survey has been an efficient data collection approach for understanding the social or physiological aspects of occupants. Nowadays, wearable sensors can enable studies to understand the physiological situations of occupants, etc., so as to make a cross-validation with the subjective respondents and enhance the accuracy and reliability of the research.

Therefore, it is necessary to conduct a comprehensive review of the applications of sensing technologies in the investigation of HBI from the perspective of the human-building-environment system, including the occupant, the building system, the environment, as well as the building consumption, to provide more references for data acquisition methods for future HBI studies. This article aims to conduct a systematic review of the sensing technologies employed by pertinent HBI research publications. The study is divided into three main sections, with Section 1 introducing the context and purpose of the review, Section 2 introducing the review criteria and methodology, Section 3 presenting the thorough review results, Section 4 discussing the results, and Section 5 providing a conclusion.

2. Materials and Methods

This study conducted a systematic review of the sensing technologies as a data acquisition method in the publications relevant to HBI between the years 2012 and 2022, adhering to the PRISMA guidelines (Figure 1). The Web of Science database was chosen for this study because it is one of the most frequently used academic search engines. The terms “sensor”, “sensing”, and “sensing technology” were used to collect studies on sensing technology. HBI-related search terms such as “human building interaction”, “occupancy”, “occupant behaviors”, and “energy behavior” were also included. Additionally, construction building technology, civil engineering, engineering electrical electronics, and instrument instrumentation were selected as academic study areas. According to Figure 1, finally, 120 documents were remaining after identification, abstract reading, and full paper reading processes for in-depth analysis and classification. As shown in Figure 2, the papers that were reviewed were divided into six categories based on the research topics related to HBI: (1) occupancy status sensing; (2) sensing of occupant physiological indicators; (3) sensing of building component; (4) sensing of environmental conditions; (5) sensing of building consumption; and (6) application of the fusion of the multi-sensing system.

3. The Sensing Technologies Adopted by HBI Research

3.1. The Sensing of Occupancy Status

The occupancy status in the ASHRAE handbook-HVAC application covers five aspects: presence, count, position, trajectory, and action [10]. Image-based sensing, infrared-based sensing, radio frequency-based sensing, and ultrasonic sensors are the four primary sensing technologies used to study occupancy status at present.

3.1.1. Image-Based Sensing

In HBI research, image-based sensing technologies are widely utilized to monitor building occupancy information and collect occupancy-related data. Choi et al. [10] grouped image data into three basic categories: RGB(R), depth (D), and thermal (T). The references reviewed for imaged-based sensing technologies are shown in Table 1.

(1)

The RGB image

The RGB image is supported by the RGB color model, in which the three primary colors, red, green, and blue, are blended in a variety of ways to create other hues. Due to their low cost and user-friendliness, web cameras and surveillance cameras are the most commonly utilized equipment for acquiring RGB images [11,12,13,14,15]. The RGB image taken by the webcam will be converted into numeric variables for further processing. In their studies, Yoon et al. [11] employed a webcam to capture images of occupant activities and transformed them into quantitative variables for classification algorithm analysis. Yang et al. [16] employed an overhead camera installed in the lecture room to capture students’ actions and determined the number of students who entered and exited using the vision algorithm. In Ref. [17], the authors employed Azure Kinect to classify the garments in CLO values to investigate the relationship between clothing insulation and occupant thermal comfort, and in Ref. [18], the authors continued the research by implementing the AI model in a connected thermostat system.

(2)

The depth image

Depth images can be acquired by a depth camera. Structured light, binocular vision, and time of flight (TOF) are now the most used approaches to depth perception [19]. According to the fundamental principle of structured light, the controllable light is projected onto the surface of the item to be measured and then captured by an image sensor [20]. Structured light technology has certain benefits, including lower cost, lower power consumption, less influence by ambient light, and a quick response time, but the long-distance error also grows quickly. Using a triangle ranging method, binocular stereo-vision imaging calculates the distance between the measured object and the two installed cameras. The binocular vision method is adaptable to outdoor environments, but its image quality is sensitive to ambient light and dependent on the calculation’s complexity [21]. In comparison to structured light and binocular vision, TOF is currently the least susceptible to environmental light, with the fastest response speed and maximum accuracy, but at the largest expense. TOF works by transmitting a constant stream of light pulses to the target and then using a sensor to detect the light that is reflected back from the target in order to establish the target’s distance [22]. Several depth camera products, including Microsoft Kinect V2 and HoloLens, have adopted TOF.

Several studies have employed depth image sensing technology to track the activities of building occupants [23,24,25,26]. In the study by Dziedzic et al. [24], Microsoft Kinect was used to capture both RGB and depth images, which were then processed into color data and human skeleton data, enabling the acquisition of the occupant’s movement map, identification, activity estimation, clothing insulation, and the formation of an occupant indoor profile. Additionally, Na. et al. [23] employed Kinect in conjunction with a deep learning algorithm to predict the metabolic rate of occupants. Lu et al. [27] created a low-resolution TOF sensor system for accurate occupancy accounting in commercially-built environments that could accommodate several individuals walking in unpredictable directions.

The advantage of image-based sensing technologies being true and accurate is providing the ground truth of occupancy information from various aspects, including occupant presence, occupant activity, occupant tracking, occupant location, as well as occupant counting; therefore, image-based sensing is often used as a validation method for other sensing technologies. Currently, most of the public and commercial buildings have been installed with surveillance cameras for safety monitoring, which is a convenient way to capture occupancy images, but it will also raise privacy issues and is not suitable for the investigations involved with the residential buildings. Except for the TOF-based depth image, which is without lighting requirement, the acquisition of RGB image and another two types of Depth image all have requirements of clear and unobstructed field views and good lighting conditions. When processing image data, this normally demands a high computational power associated with a high cost of computation hardware. For the image processing techniques, deep learning-based CNN models are relevantly commonly used in the studies, referring to [17,18,23,24]; and for the studies using the dynamic video as research materials to extract occupancy information, the background subtraction algorithm is a popular method for moving target recognition, as in Refs. [16,25]. Additionally, as depth image sensing can simultaneously capture the RGB and depth information and calculate the distance from the object to the camera, except for the image processing, the methodologies of point cloud clustering and 3D reconstruction, as well as the human skeleton model, are also feasible techniques in studies to investigate occupant activities, as shown in [23,25,27].

Table 1. The list of references for image-based sensing technologies.

Ref.	Type	Year	Research Object	Sensing Device	Data Processing Method
[11]	RGB	2022	Occupant activities	Web camera	SVM, KNN, RF, manually labeling
[12]		2018	Occupant tracking	View camera	rHOG
[13]		2017	Occupant presence	Web camera	stochastic model
[14]		2017	Occupant positions	Camera-based indoor tracking system	localization algorithms, calibrated mapping algorithm
[15]		2022	Occupancy counting	Web camera	Statistical analysis, K-means clustering, multiple linear regression
[16]		2018	Occupancy counting	overhead video; PTZ camera	Background Subtraction algorithm, SVM + HOG
[17]		2022	Occupant clothing insulation	Azure Kinect	CNN models-VGG 16, Inception V4, TinyYOLOV3, ResNet18
[18]		2022	Occupant clothing insulation	Video camera	CNN-YOLO models
[23]	Depth	2020	Occupant activity	Microsoft Kinect	CNN model-METNet
[24]		2019	Occupant activity	Microsoft Kinect	depth registration; skeleton model, CNN
[25]		2015	Occupancy detection and profiling	Microsoft Kinect MESA SR4000	background subtraction, point cloud clustering
[26]		2017	Occupancy counting	Microsoft Kinect	FORK
[27]		2021	Occupancy counting	VL53L5TOF sensor	3D reconstruction, Background subtraction, and filtering, point clustering

Table 1. The list of references for image-based sensing technologies.

Ref.	Type	Year	Research Object	Sensing Device	Data Processing Method
[11]	RGB	2022	Occupant activities	Web camera	SVM, KNN, RF, manually labeling
[12]		2018	Occupant tracking	View camera	rHOG
[13]		2017	Occupant presence	Web camera	stochastic model
[14]		2017	Occupant positions	Camera-based indoor tracking system	localization algorithms, calibrated mapping algorithm
[15]		2022	Occupancy counting	Web camera	Statistical analysis, K-means clustering, multiple linear regression
[16]		2018	Occupancy counting	overhead video; PTZ camera	Background Subtraction algorithm, SVM + HOG
[17]		2022	Occupant clothing insulation	Azure Kinect	CNN models-VGG 16, Inception V4, TinyYOLOV3, ResNet18
[18]		2022	Occupant clothing insulation	Video camera	CNN-YOLO models
[23]	Depth	2020	Occupant activity	Microsoft Kinect	CNN model-METNet
[24]		2019	Occupant activity	Microsoft Kinect	depth registration; skeleton model, CNN
[25]		2015	Occupancy detection and profiling	Microsoft Kinect MESA SR4000	background subtraction, point cloud clustering
[26]		2017	Occupancy counting	Microsoft Kinect	FORK
[27]		2021	Occupancy counting	VL53L5TOF sensor	3D reconstruction, Background subtraction, and filtering, point clustering

3.1.2. Infrared-Based Sensing

Active infrared sensing and passive infrared sensing are the two categories of infrared-based sensing. The beam-break sensor, as a typical active infrared sensing approach, has been employed by previous investigations to estimate occupancy counting [28]. The beam-break sensor is made up of a transmitter and a receiver. The infrared light-emitting diode of the transmitter emits a modulated infrared light beam that is received by the infrared photoelectric sensor of the receiver, which converts the optical signal into an electric signal.

Without actively emitting infrared light, the passive infrared sensor (PIR) detects and collects the infrared radiation emitted by the human body through its pyroelectric elements, then converts the infrared heat into electric signals. PIR sensors have been the predominant sensing technology in investigations of occupant behavior, with pertinent research areas including occupancy counts, localization, and motion detection [29,30,31,32,33,34]. However, the PIR sensor also has the issue that it is ineffective if the occupant is stationary.

Passive infrared sensing is also a fundamental component of the thermal imaging camera. The thermal camera is an integration of an optical system, an infrared thermal sensor, and an electronic system that can detect, process, and transform infrared radiation into thermal images by showing the temperature distribution on the camera’s screen. As a non-invasive and privacy-friendly device, the thermal camera has been employed by a number of studies to measure the thermal comfort of occupants [35,36], as well as to estimate the state of occupancy [37,38,39]. Li et al. [40] created a deep-learning model to predict thermal comfort based on age, gender, and human skin temperature extracted from thermal pictures. The list of papers reviewed for infrared-based sensing is shown in

Table 2. The list of references for infrared-based sensing technologies.

No.	Type	Year	Research Object	Sensing Device	Data Processing Method
[28]	Active	2017	People counting	beam-break sensor	thresholding algorithm
[29]	Passive	2016	Occupancy presence	PIR sensor	Hidden Markov models
[30]		2021	Occupancy presence	Infrared Sensor Array	Image processing, multi-Bernoullii filter
[31]		2021	Occupancy pattern	PIR sensor	deterministic modelling
[32]		2018	Occupant tracking	PIR sensor	accessibility map, A-Star algorithm
[33]		2022	Occupant location and activity intensity	PIR sensor	SVM
[34]		2019	Occupancy presence	PIR sensor	Find state algorithm
[36]	Thermal camera	2018	human skin temperature	Thermal camera	Thermal image processing software
[37]		2021	Occupancy estimation	Thermal imaging sensor	Blob extraction algorithm, blob filtering algorithm, KNN, SVM, RF
[38]		2019	Occupancy estimation	Thermal camera	DNN model
[39]		2021	Occupancy counting	Thermal camera	U-Net-like CNN
[40]		2022	Occupant thermal comfort	Thermal camera	CNN model

.

Both the beam-break sensor and the PIR sensor have the advantages of low cost, simple installation, and being friendly to privacy because they only output binary data (0 for absence and 1 for presence). In the current market, PIR sensors are used to activate a number of smart home appliances; however, PIR sensors have one obvious drawback that many users have complained about: the detection of occupancy presence by PIR sensors depends on the movement of the occupant, and if the occupant keeps stationary for even a moment the PIR sensor will make a false absence detection and turn off the smart domestic appliances, which have caused many problems for the users. For the Break beam sensor, its limitation lies in that its accuracy is reduced when multiple people pass simultaneously. The thermal camera can perceive the heat radiation emitted by an occupant and create a thermal image, so it is also classified into the group of image-based sensing techniques in the review of Choi et al. [10]. Similar to image-based sensing, thermal camera sensing also require a clear and unobstructed field of view, but it does not need good lighting conditions as it works well in the darkness and the deep learning-based models for image processing are also applicable for thermal images. Additionally, the thermal camera can be integrated with developed thermal image processing software, refered to in study [36], which provides support to the research.

3.1.3. Radio Frequency Signal-Based Sensing

Table 3 shows studies involved with the application of radio frequency signal-based sensing technologies. According to Ref. [41], by measuring the proximity, the distance, and the distortion of the signals, radio signal sensing has been able to provide supporting information to predict the occupancy status, including the location, presence, count, identity, and trajectory. The current radio technologies that have been used for occupancy detection include Wi-Fi, Bluetooth low energy (BLE), radio frequency identification (RFID), ultra-wideband (UWB), and global positioning system (GPS).

Wi-Fi is currently extensively available in various buildings. Numerous studies have demonstrated that it is reasonable to estimate occupancy counting based on the number of Wi-Fi connections, given that connecting to Wi-Fi with a smartphone or tablet to access the internet has become a habit of occupants [42,43,44]. Alishahi et al. [45] stated the Poisson regression model they proposed to extract key occupancy indicators could perform better than the linear regression model. In Ref. [15], the authors also adopted a camera to acquire ground truth image data as a supplement to the Wi-Fi connection data, aiming to reduce the deviation between the Wi-Fi connection and the actual occupancy counting.

BLE, which is based on Bluetooth technology, provides a considerable advantage in terms of cost and power efficiency and is built expressly for IoT devices’ features and applications. As it maintains communication and transmits data at extremely low levels of power, users can search and connect quickly. BLE has been utilized in research pertaining to occupancy detection. Barsocchi et al. [46] proposed an occupancy detection system based on BLE. In the trial, they deployed a few Bluetooth receivers and assigned each participant a Bluetooth tag that was integrated with the building’s badge; the results revealed that the BLE-based solution could, at most, enhance occupancy detection accuracy by 10%. Tekler et al. [47] have developed a strategy to capture the mobility pattern of occupants using BLE technology and an ensemble clustering model and verified the model through a five-week case study in a Singapore office building.

RFID is a technique for automatically identifying and tracking objects and people through the use of tags with a unique code. RFID has been widely implemented in the construction phase of prefabricated buildings, when building components or construction materials are manufactured, transported, and assembled in an industrialized manner [48]. During the building’s operation phase, RFID has been utilized by many studies to investigate the occupancy status. The authors of Ref. [49] presented an RFID-based occupancy detection system to enable demand-driven HVAC operations by simultaneously detecting and tracking several stationary and moving people in multiple locations. Recently, Kong et al. [50] also published an article introducing an experimental study in which the authors developed an occupant-centric control of the HVAC system by integrating a depth camera, pressure sensor, and RFID sensor (Figure 3), and reported that more than 80% of the occupants were satisfied with the performance of the control system, as well as achieving 17% weekly average energy savings.

GPS and UWB are both positioning technologies. UWB is a short-range RF method for wireless communication that can precisely pinpoint people, equipment, and assets. UWB is predominantly used in the indoor environment for occupant detection [51], for instance, in Refs. [52,53], UWB technology was used to construct a human detection system in the smart home setting. GPS is well-known to the general public as a navigation technology and for capturing the movement trajectory of users, and it could also be considered as a non-intrusive sensor to locate the occupant in the building [54,55]. However, the authors pointed out its low accuracy in comparison to other occupancy sensing methods [54].

This section reviews five sensing technologies based on radio frequency signals. The five sensing technologies share the advantages of low cost and low power consumption; however, each sensing technology has its own concerns that require special consideration. Because numerous commercial and public buildings have already been deployed with a Wi-Fi network, it will be convenient to use Wi-Fi signals to acquire occupancy status information, however, the problems associated with this method cannot be neglected. In practice, some people may carry several smart products connecting to Wi-Fi, while there are also some people who neither take any devices nor connect to the internet; as a result, the occupancy based on the Wi-Fi connection might output false consequences. Additionally, in order to improve the security mechanism and protect user privacy, many device manufacturers have developed a random MAC address function, which means the MAC addresses of the smart devices that are detected by the Wi-Fi probe are from random generation, leading to a false data matching of the occupants who use the Wi-Fi. Although BLE is quite extensive in smart devices today, its detection accuracy could be sacrificed as occupants may not enable the Bluetooth function of their devices. RFID can accurately monitor the location and identification of occupants, but its signals are susceptible to interference from metallic objects. In addition, if RFID is used for occupancy detection, each occupant must wear an RFID badge, or else the accuracy of the detection will be compromised. UWB could provide indoor occupant positioning with greater precision than Wi-Fi and BLE, and because it utilizes the excess bandwidth, it has the lowest power requirement. Due to its prospective applications, it has been incorporated into a variety of smart devices, including the iPhone, iPad, and Samsung’s Galaxy smartphones. The tags for UWB-based location and pairing systems are more expensive than Bluetooth and RFID tags, and due to its low data transmission rate, it cannot replace Bluetooth and Wi-Fi technologies for large data transfers, making it impractical for streaming large amounts of data. In comparison to four other radio frequency signal-based sensing techniques, GPS is primarily used for outdoor positioning due to non-line-of-sight communication issues that prevent GPS from providing accurate interior location data. The two studies in Refs. [54,55] use GPS data from users’ devices to detect whether they have entered the building or not, but does not provide specific occupancy information.

Table 3. The list of references for radio frequency signal-based sensing.

No.	Type	Year	Research Object	Data Processing Method
[42]	WIFI	2019	Occupancy counting	Multiple linear regression, ANN
[43]		2022	Occupant behavior	lightweight CNN
[44]		2019	Occupancy detection	Ensemble learning classification algorithms
[45]		2021	Occupancy pattern	K-means clustering, Poisson regression, cumulative frequency analysis
[46]	BLE	2017	Occupancy detection	SVM, RF
[47]		2020	occupancy pattern	Binary classification, gradient boosting algorithm, K-means algorithm,
[49]	RFID	2012	Occupancy counting, occupancy identification	Scattering analysis, statistical analysis
[50]		2022	Occupancy counting	Radio signal processing
[51]	UWB	2017	Occupancy detection	Principal component analysis (PCA)
[52]		2017	Human identification	Region of interest extraction, PCA
[53]		2021	Motion detection	adaptive motion detection algorithm
[54]	GPS	2021	Occupancy counting	GeoHash Model
[55]		2021	Occupancy schedule	Web scraping techniques, text classification, and semantic analysis

Table 3. The list of references for radio frequency signal-based sensing.

No.	Type	Year	Research Object	Data Processing Method
[42]	WIFI	2019	Occupancy counting	Multiple linear regression, ANN
[43]		2022	Occupant behavior	lightweight CNN
[44]		2019	Occupancy detection	Ensemble learning classification algorithms
[45]		2021	Occupancy pattern	K-means clustering, Poisson regression, cumulative frequency analysis
[46]	BLE	2017	Occupancy detection	SVM, RF
[47]		2020	occupancy pattern	Binary classification, gradient boosting algorithm, K-means algorithm,
[49]	RFID	2012	Occupancy counting, occupancy identification	Scattering analysis, statistical analysis
[50]		2022	Occupancy counting	Radio signal processing
[51]	UWB	2017	Occupancy detection	Principal component analysis (PCA)
[52]		2017	Human identification	Region of interest extraction, PCA
[53]		2021	Motion detection	adaptive motion detection algorithm
[54]	GPS	2021	Occupancy counting	GeoHash Model
[55]		2021	Occupancy schedule	Web scraping techniques, text classification, and semantic analysis

3.1.4. Ultrasonic-Based Sensing

The ultrasonic sensor determines the target’s distance by producing ultrasonic sound waves and converting the reflected sound into an electrical signal. It consists of two parts: the transmitter, which employs piezoelectric crystals to generate sound, and the receiver, which detects the sound after it has traveled to and from the target. In related HBI research, the ultrasonic sensor was used to detect the presence of occupancy or to count the number of occupants. Shin et al. [56] described a method for person recognition and counting that employed ultrasonic signal transmission into a room and the analysis of the superposition of the reflections caught by a microphone. In Refs. [57,58], the authors presented their ultrasonic chirp-based research on occupant number estimation and indoor location tracking, respectively. Ghosh et al. [59] installed ultrasonic sensors in two distinct deployments as sensor grids and placed them on the door-frame in order to identify human activity in a smart home environment. They reported a detection accuracy of more than 90 percent for different activities, including sitting, standing, and falling.

The disadvantages associated with ultrasonic sensors also need to be paid attention to, according to Refs. [56,57]; as an active sensing system, it is necessary to have a mechanism to coordinate signal transmissions in case multiple sensors are installed in the same area to prevent the occurrence of cross-talking. In larger spaces, a proportionally powerful transmitter is required, accompanied by a bigger amplifier and transducer to accommodate the increased power, which leads to it being increasingly difficult to determine the precise number of individuals present due to the blurring of individual identities. The list of references for ultrasonic-based sensing is shown in

Table 4. The list of references for ultrasonic-based sensing.

No.	Type	Year	Research Object	Sensing Device	Data Processing Method
[56]	Ultrasonic sensor	2016	Occupancy detection, counting	wide-band ultrasonic transmitter, ultrasonic MEMs microphone	Semi-supervised learning model, classification, regression trees;
[57]		2015	Occupancy detection, counting	Motu Ultra-Light MK3 DAC and ADC, audio amplifier, omnidirectional tweeter, measurement microphone	DBSCAN algorithm, regression model
[58]		2012	Occupant location tracking	audio speakers	Signal processing-TDOA technique, pulse compression
[59]		2019	Human activity	ultrasonic sensors	Threshold-based classification

.

3.2. The Sensing of Occupant Physiological Indicators

As humans are the focus of the human-building system, the occupant’s feedback, experience, and perception of the built environment can provide useful insights to support the building design and operation. The traditional methods to solicit occupants’ feedback or evaluation about the building environment generally depend on interview and questionnaire surveys, which are convenient and straightforward but inherent with the drawbacks of subjective biases. In recent years, with the advancement of neuroscience, sensor technology, as well as the Internet of Things, emerging wearable devices have provided new inspirations for researchers to investigate the interactions between the occupants and built environments. Except for the subjective self-reporting data, plenty of studies have employed wearable devices equipped with a variety of biosensors to measure the objective physiological feedback, followed by cross-validation based on both subjective and objective data, which can provide stronger evidence for their conclusions. In the current publications, biosensing techniques can be classified into four broad categories, including brain activity-related, muscle and skin-related, and heart-related [60]. Table 5 is the list of studies with physiological measurements in this review.

3.2.1. The Sensing of Human Brain Activity

The biosensing techniques to assess brain activities mainly include electroencephalogram (EEG), event-related brain potentials (ERPs), functional magnetic resonance imaging (fMRI), positron emission tomography (PET), and magnetoencephalograms (MEGs) [60]. EEGs have been widely applied in the research of human behaviors in built environments. EEG is a brain imaging technology employing scalp electrodes to monitor voltage variations caused by brain neuronal activities. The indicators of EEG rely on the power densities of distinct frequency bands, which are connected to arousal, concentration, stress, and a variety of other mental activities [61]. In previous studies, authors utilized EEG to assess the mental states of the occupants to further analyze the influence of the built environment on the occupants as well as to conduct a post-occupancy evaluation [61,62,63]. Aiming to evaluate the design of the classroom before construction, Cruz-Garza et al. [64,65] asked participants to wear an EEG device and VR glasses while completing cognitive tasks in virtual classroom environments of different design conditions. The authors found a significant difference in EEG features across participants in the different classroom design schemes when the cognitive tasks involved short-term memory, which may provide implications to designers in which a design scheme could create a more friendly space for students to study or to finish the tasks, as a methodology for pre-construction evaluation.

Currently, the use of EEG in HBI-relevant studies is primarily focused on two aspects: post-occupancy evaluation and pre-construction evaluation of design plans. In post-occupancy evaluation studies, the EEG is used to assess occupants’ responses to the existing indoor environment of the building, particularly in terms of thermal comfort. Researchers have attempted to determine the relationships between EEG signals and thermal comfort or thermal sensation in humans, and their research has shown that the EEG signal is intimately linked to sensory and cognitive processes that are affected by external stimuli such as temperature, relative humidity, and air velocity [62,63]. In design evalution, during the pre-construction period, virtual reality technology is typically used as a tool for the 3D visualization of the design plan, and experimental subjects are asked to wear VR glasses to immerse themselves in the virtual design scheme while their EEG signals are assessed simultaneously, in order to determine the effect of the design on the building user’s mental state, emotion, cognitive competence, etc. [64,65]. Typically, the data processing procedures of EEG data consist of three steps: data pre-processing with Matlab toolbox, feature extraction for the EEG frequency band, and machine learning-based EEG pattern recognition; SVM, KNN, RF, and DT are prevalent ML algorithms utilized in studies. In addition, cross-validation requires a questionnaire to capture respondents’ subjective opinions about the building environment.

3.2.2. The Sensing of Muscle and Skin Activity

Electromyography (EMG), electrodermal activity (EDA), and skin temperature (ST) are the most common biosensing technologies used to measure the responses of human muscle and skin. The EMG technique is a measurement of the peripheral nervous system by recording the electrical activity produced by skeletal muscles, allowing the researchers to discover some subtle muscle movements and assessing the muscle load or forces during the engagement of some labor tasks [66]. EDA, also named as the galvanic skin response (GSR), measures the electrical changes of the skin through sweat secretion and has been discovered to have a substantial relationship with human emotional arousal [67]. Higher arousal levels are associated with greater skin conductance, so EDA can reveal how strong an emotion is. Skin conductance is subject to regulation by the human body via the sympathetic nervous system, and the electrodes of EDA devices could record the differences in electrical activity produced by sweat glands as they change activity. Skin temperature (ST) is an essential indicator to monitor the human body’s thermoregulatory system, which can be measured by a variety of thermometers [68]. By reviewing past studies, the EMG has been primarily utilized in indoor safety-related topics of building operation, such as to investigate the physical demands of firefighting personnel in the emergency evacuation of high-rise buildings [69,70]. The EDA and the ST are two common approaches to monitoring the indoor thermal comfort conditions of occupants, which have been applied in many studies [71,72,73,74].

3.2.3. The Sensing of Heart Activity

Electrocardiogram (ECG) monitoring is the typical heart biosensing technology. ECG is to monitor the health of the heart by measuring the duration and intensity of electrical waves traveling through the heart. The conventional ECG monitor employs wet electrodes to capture the electrical signal of the heartbeat and calculates three parameters, including the heart rate (HR), the time interval between individual heartbeats (abbreviated as IBI), and the fluctuation of IBI, referred to as HRV [75]. Zhu et al. [76] recorded the ECG data of 6 subjects under 60 indoor environments and revealed the relationships between HRV and thermal sensations of occupants.

HRV is a physiological indicator commonly used to assess levels of thermal comfort. The results of several studies have shown that LH/HF (ratio of the low-frequency power and high-frequency power of the HRV analysis results) can be influenced by factors including the variations in ambient temperature, humidity, airspeed, as well as occupants’ thermal sensations [75,76]. Subjects will feel thermally comfortable when LF/HF is approximated to 1, and higher values have occurred during uncomfortable states, while low values occurred during relaxed states.

3.2.4. The Wearable Device

The popularity of health-monitoring wrist-worn wearable devices, often known as smart wristbands or smart watches, has increased among customers. The smart health bracelet typically includes two sensors to monitor the status of the heart: an electrocardiogram (ECG) sensor and a photoplethysmography (PPG) sensor. The PPG sensor provides the heart rate data, which is a light-based technology that delivers green light to the skin and then detects periodic changes in light intensity caused by blood circulation [77]. Compared to the ECG, the accuracy of PPG is liable to be influenced by several factors, including wrist placement, the dominant hand, the body, environmental temperatures, etc. [68].

Until recently, smart wristband solutions have been able to acquire multi-physiological data by combining many biosensors, allowing for more thorough monitoring of human physiological status. In recent publications, in order to get more objective data to study the physiological status of the subjects, the use of many wearable devices simultaneously or a single multifunctional smart wristband has been used [73,75,78]. In Ref. [79], the authors used two wearable devices(Figure 4) to assess four essential physiological signals (EEG, EDA, ECG, ST) on 52 participants in a controlled setting under three distinct thermal conditions (cold, warm, and neutral) in an effort to give further evidence for the research of the personal comfort model (PCM). The findings of their experiments suggested that physiological measurements can detect specific temperature sensations, which is essential for the development of the most advanced PCMs and the discovery of fresh energy-saving choices that account for individual variances.

In the HBI research, a greater comprehension of how people react to the built environment will aid in designing and controlling for occupant reactions. The invention of wearable sensors has greatly facilitated the investigation of building occupants’ responses to their environment. Some wearable sensor products not only make it easier to conduct experiments, but they can also provide a platform for data processing and analysis, which has greatly aided research. Nonetheless, the limitations associated with this platform must be emphasized, as indicated by Ref. [64], which utilized the wearable sensor product Emotiv EPOC for data collection. Emotiv lacks open-source software and is difficult to convert to MATLAB; furthermore, Emotiv’s algorithm for data analysis is a “black box”, so in their study, they could only examine visual variables. In addition, for EEG measurement, wearable devices reduce the number of electrodes to reduce user burden and eliminate the need for conductive gel, which is used in conventional devices. However, there is a trade-off between signal quality and the number of electrodes in any EEG system, therefore, the loss of sensor accuracy in such wearable devices must be acknowledged.

In addition, the majority of research findings in relevant studies were based on a controlled environment, which is vastly different from the actual world; for instance, in Refs. [64,65], the experimental environment was set up using VR technology, and the subjects were stimulated only by the immersive virtual environment with controlling environmental factors. However, all other real-world environmental factors were not accounted for in the experiment; thus, the applicability of this knowledge to real-world settings remains uncertain.

Table 5. The list of references for occupant physiological indicators sensing.

No.	Type	Year	Research Object	Sensing Device	Data Analysis Method
[61]	EEG	2018	Brain activity in rest and task	Emotiv EPOC	EEGLAB toolbox LDA classifier
[62]		2019	Brain activity in rest and task	Emotiv EPOC	EEGLAB toolbox LDA, SVM
[64]		2020	Brain activity in VR environment	Emotiv EPOC	software Emotiv Pro, statistical analysis
[65]		2022	Brain activity in VR environment	63-channel actiCHamp	Lab Streaming Layer (LSL) software
[66]	EMG IMU	2021	Worker’s muscle engagement	Myo armband	ANN
[69]		2014	Physical demand	Delsys wireless EMG system	MATLAB, statistics analysis
[70]		2021	Leg fatigue, gait motion	Megawin, Qualisys Track Manager	MATLAB, statistic analysis
[71]	EDA, ST	2019	Environmental comfort	Careshine Electronic Technology, PyroButton-L	Statistic analysis
[72]		2019	Skin temperature, face temperature	Wearable device with infrared temperature sensor, thermal camera	Neighborhood component based feature selection, RF, SVM, KNN
[73]		2022	Occupant thermal comfort	E4 Wristband	CNN-SVM hybrid model, ensemble transfer learning
[74]		2018	Occupant thermal comfort	Exacon D-S18JK	Statistical analysis, SVM, ELM
[75]	ECG	2020	Environmental comfort	EPOC+, BioHarness	Feature extraction, LDA, KNN, decision Tree, naïve Bayes, SVM, and RF
[76]		2018	Occupant thermal comfort	Holter	Statistic analysis
[77]	PPG	2019	Pervasive blood pressure	Smart wristbands	Feature extraction, NN, SVM, DT
[78]	ECG, EEG, EMG, GSR	2022	Indoor thermal comfort	Physiological signal measurement system	Linear regression, Gaussian process regression, SVM regression, DT
[79]	EEG, EDA, BVP, IBI, ST	2022	Thermal comfort	MUSE 2 headband Empatica E4	Feature extraction, statistic analysis

Table 5. The list of references for occupant physiological indicators sensing.

No.	Type	Year	Research Object	Sensing Device	Data Analysis Method
[61]	EEG	2018	Brain activity in rest and task	Emotiv EPOC	EEGLAB toolbox LDA classifier
[62]		2019	Brain activity in rest and task	Emotiv EPOC	EEGLAB toolbox LDA, SVM
[64]		2020	Brain activity in VR environment	Emotiv EPOC	software Emotiv Pro, statistical analysis
[65]		2022	Brain activity in VR environment	63-channel actiCHamp	Lab Streaming Layer (LSL) software
[66]	EMG IMU	2021	Worker’s muscle engagement	Myo armband	ANN
[69]		2014	Physical demand	Delsys wireless EMG system	MATLAB, statistics analysis
[70]		2021	Leg fatigue, gait motion	Megawin, Qualisys Track Manager	MATLAB, statistic analysis
[71]	EDA, ST	2019	Environmental comfort	Careshine Electronic Technology, PyroButton-L	Statistic analysis
[72]		2019	Skin temperature, face temperature	Wearable device with infrared temperature sensor, thermal camera	Neighborhood component based feature selection, RF, SVM, KNN
[73]		2022	Occupant thermal comfort	E4 Wristband	CNN-SVM hybrid model, ensemble transfer learning
[74]		2018	Occupant thermal comfort	Exacon D-S18JK	Statistical analysis, SVM, ELM
[75]	ECG	2020	Environmental comfort	EPOC+, BioHarness	Feature extraction, LDA, KNN, decision Tree, naïve Bayes, SVM, and RF
[76]		2018	Occupant thermal comfort	Holter	Statistic analysis
[77]	PPG	2019	Pervasive blood pressure	Smart wristbands	Feature extraction, NN, SVM, DT
[78]	ECG, EEG, EMG, GSR	2022	Indoor thermal comfort	Physiological signal measurement system	Linear regression, Gaussian process regression, SVM regression, DT
[79]	EEG, EDA, BVP, IBI, ST	2022	Thermal comfort	MUSE 2 headband Empatica E4	Feature extraction, statistic analysis

3.3. The Sensing of Building Components

3.3.1. Magnetic Reed Switch

The interactions between people and a building can be predicted and modeled using data collected through real-time monitoring of building components, such as windows, doors, and shutters. In certain publications, the sensing technology that can detect the states of building components is categorized as threshold and mechanical sensors, with the magnetic reed switch being the most commonly used sensor [41,80,81].

The magnetic reed switch is actuated by magnetism. The reeds consist of two thin, flexible, ferromagnetic metal blades that are hermetically encased in a glass bubble. The magnetic reed switches offer low cost, low power consumption, and simple installation. Reed contacts can detect whether a window or door is open or closed after adhering to the frames; however, they are limited in that they cannot measure the window’s or door’s degree of opening; furthermore, they cannot detect the slightest change in window/door status, which could miss significant implications [41]. The magnetic reed switch sensor can be found in numerous articles pertaining to the modeling of window/door opening behavior, the influential elements of ventilation behavior, and the consequences of ventilation behavior on indoor air quality [82,83,84,85,86,87].

Due to the magnetic materials in the door or window sensor, as long as a strong magnetic field is present, the door and window sensor may be incorrectly interpreted as closed. And if the doors are made of iron, careful implementation consideration should be given to the location of the sensor, as iron doors have a significant attenuation effect on wireless signals. If the door sensor is placed incorrectly, it may be unreliable or even unable to communicate with the gateway. It is recommended that door sensors are not installed on iron security doors, as this can interfere with communication and cause the iron door to become magnetized by the sensor. To ensure communication, the gateway must be as close as feasible to the door sensor, and vibrations can also cause door and window sensors to misinterpret their surroundings, which needs to be paid attention.

3.3.2. The Vibration Sensor

The vibration sensor is an electronic device that can detect and measure the amplitude and frequency of vibration in a system or piece of equipment, which can accurately record the mechanical vibration quantity, including the velocity, displacement, and acceleration, then convert them into electrical signals for the outputs. In the research of human building interaction, vibration sensors have been commonly installed on the floor to detect footstep-induced structural vibrations, then, by applying the signal processing technique and machine learning algorithm, the researchers could investigate occupancy counting [88], occupant detection [89,90,91], occupant localization [92,93,94], and occupant activity level [95]. Figure 5 depicts the framework established by Drira et al. [94] for occupant detection, localization, and tracking using footstep-induced floor vibrations.

The limitations of structural vibration sensors should be addressed, according to the review. The first challenge is that the vibration caused by the ambient environment may be more obvious than the vibration caused by the footsteps, resulting in false detection of the vibration sensor. The second challenge is that in practice, other vibrations, such as the movement of equipment, may be similar to the footstep, resulting in misjudgment.

Table 6. The list of references for building components sensing.

No.	Type	Year	Research Object	Sensing Device	Data Processing Method
[82]	Magnetic reed switch	2020	Window state	Window sensor	Statistic analysis
[83]		2018	Occupants’ window behavior	Window sensor	Statistic analysis
[84]		2015	Occupants’ window behavior	Window sensor	Monte Carlo simulation
[86]		2015	Monitor elderly’s behavior	Door contact	activities assessment algorithm developed by authors
[87]		2018	Occupants’ ventilation habits	Window sensor	Statistic analysis
[88]	Vibration sensor	2018	Occupancy counting	Geophone SM-2	Detection algorithm
[89]		2020	Occupant detection	Geophone SM-2	Transfer learning
[90]		2021	Occupant detection	Vibration sensor	CWT, SVM, CNN, finite element simulation
[91]		2016	Occupant detection	Vibration sensor	a two-stage step-induced signal detection algorithm
[92]		2018	Occupant localization	Geophone, amplifier	Anomaly detection algorithm, SVM
[93]		2019	Occupant Tracking	Vibration sensor	Signal processing, error-domain model-falsification
[94]		2022	Occupancy detection and tracking	Vibration sensor	Signal processing, error-domain model-falsification
[95]		2019	Occupant activity	Geophone, amplifier, ADC module, Raspberry Pi	Signal processing, noise filtering, vibration detection.

shows the papers reviewed for the sensing of building components in this article.

3.4. The Sensing of the Building Environment

The indoor and outdoor building environmental factors are significant to the research topics of human–building interaction, such as the influential factors of occupant behaviors in the building [83,96], the modeling of occupant behavior patterns [97,98,99], or the effects of occupant behaviors on the indoor environment [100], etc. Various environmental sensors have been utilized in past research as an efficient method for the collecting of environmental data, as shown below in Table 7.

3.4.1. Air Property Sensor

(1)

CO₂ sensor

The CO₂ sensor measures the concentration of CO₂ in the atmosphere. Many studies have demonstrated that CO₂ concentration is an effective indicator for detecting and estimating occupancy states. Li et al. [101] found the strongest association between the number of occupants and the CO₂ concentration. Dedesko et al. [102] devised a method to estimate the occupant activity and occupancy level of a hospital based on the combination of beam-break sensor data and CO₂ concentration data. Wolf et al. [103] offered a workflow (Figure 6) illustrating the method for estimating the occupancy state based on CO₂ level.

(2)

Indoor air quality (IAQ) monitor

The IAQ monitor is a multifunctional integrated sensing device to measure a variety of indoor environment quality indicators, including temperature, humidity, the concentration of PM2.5, CO₂, TVOC, and other air pollutants. According to the design of the research, the IAQ monitor can be adjusted to integrate the appropriate sensors. The study [104] employed an IAQ monitor measuring indoor temperature, relative humidity, and CO₂ concentration to investigate the impact of window operation behaviors on the simulated energy performance in university residence halls. Jia et al. [105] integrated three distinct sensors, a CO₂ sensor, an illumination sensor, and a temperature/humidity sensor (Figure 7) in order to collect environmental data and test the occupant behavior model in a commercial building. Kim et al. [106] merged the data received in real-time by four types of environmental sensors—a temperature/humidity sensor, CO₂ sensor, TVOCs sensor, and fine particulate sensor—in order to facilitate the analysis of indoor environment quality, taking occupant behaviors into account.

In some HBI studies, the CO₂ sensor is used as an indirect method to estimate the number of occupants because CO₂ is produced during human respiration. The CO₂ sensor is inexpensive, unobtrusive, and respectful of one’s privacy; however, there are some considerations to be made regarding its application. The concentration of CO₂ will be affected by the ventilation level of the building’s air, and the amount of CO₂ generated by the building’s occupants will vary greatly depending on their BMI, age, health condition, fatigue levels, etc.; therefore, the accuracy of occupancy numbers based on the CO₂ need to be combining the actual information of occupants and building operation. In addition, a CO₂ sensor is included in the IAQ monitor to measure the indoor air quality alongside humidity, temperature, TVOC, etc., which are taken as environmental factors to study the thermal comfort of occupants or the influential factors on occupant behavior. Both statistical analysis and machine learning algorithms have been used in previous research to process CO₂ concentration data.

3.4.2. Sound Sensor

The sound sensor is used to detect and monitor environmental sound waves. Current HBI research utilizes two types of acoustic sensors: the electret condenser microphone (ECM) sound sensor and the micro-electro-mechanical system (MEMS) audio sensor (Figure 8). The main component of classic voice recorders is the ECM sound sensor, in which the electret film might be vibrated by sound waves, resulting in a change in capacitance and a small voltage shift; the voltage is then collected and transferred to the computer for A/D conversion. Kim et al. [107] deployed a standard ECM-based voice recorder IVR-50 to collect voices in the home and suggested a deep learning-based sound recognition model to monitor the inhabitant’s behaviors and detect emergent occurrences in a single-person dwelling. Figure 9 depicts their study’s research procedure: (1) sound data pretreatment, (2) acoustic feature extraction, (3) classification model construction and training, and (4) results postprocessing.

MEMS is a chip-based technology integrating the micro-electronic circuit technology with a micro-mechanical system at the nano and micron scale [108]. The MEMS audio sensor incorporates both a MEMS module and an application-specific integrated circuit (ASIC) into a single package. Compared to the ECM sound sensor, the MEMS audio sensor has superior performance, smaller size, and reduced sensitivity to mechanical shocks, as well as a rapidly expanding market share in the smart technology industry, including smartphones, wearable devices, smart home, etc.; thus, the smartphone has become an effective tool for sound data collection [109]. In a study by Zhu et al. [110], the smartphone was used to detect the acoustic signal of eight energy-related occupant activities, and the locality-constrained linear coding approach was used for activity detection.

The usage of a sound sensor is a simple and low-cost technique for perceiving the acoustic environment all around. Sound recognition is also an indirect way to determine the situation of occupancy presence and to detect some specific behaviors and emergencies, such as the voice of the elderly calling for help or the elderly falling down [107], but in sound processing, some noise or an unrelevant sound must be eliminated. On the one hand, the sound environment is an important factor influencing the comfort level of building occupants.

3.4.3. Illuminance Sensor

On the basis of the photoelectric effect, the illuminance sensor measures the amount of ambient light and converts optical signals into electrical signals. In the HBI study, one key application of the illuminance sensor is to detect changes in ambient illumination in order to automatically manage the lighting system so that the indoor illuminance remains within a given range. When the room’s illumination exceeds the threshold value, the lighting system will automatically switch off or dim, and vice versa [111,112,113]. In order to determine the threshold value of indoor illuminance, Nagy et al. [112] used an illuminance sensor to record the illuminance each time an occupant manually switched the lights. They also reported that the light sensor-based occupant-centric lighting system control strategy could achieve maximum energy savings of 37.9%. Based on the continuous illuminance sensor data and occupancy survey findings, as well as the reinforcement learning algorithm, Park et al. [114] developed Light Learn, an occupant-centered lighting controller that could adapt to the preferences of the occupant and the environmental conditions.

Table 7. The list of references for building environment sensing.

No.	Type	Year	Research Object	Sensing Device	Data Processing Method
[101]	CO2	2019	Occupancy counting	CO2 sensor	Statistical analyses
[102]		2015	Occupancy counting, occupancy activity	CO2 sensor, beam-break sensor	Statistical analyses
[103]		2019	Occupancy counting	CO2 sensor	stochastic differential equations
[104]	IAQ	2017	Influence of occupant behavior on energy	IEQ monitors	Model-based simulation, statistical modeling
[105]		2019	Occupant behavior	Customized smart sensor node	Agent-based modeling
[106]		2019	Influence of IEQ on occupant behavior	SHT30, T6703 CO2 Module, SP3S-AQ2, MEVIU	weighed Euclidean distance
[107]	Sound	2020	Occupant behavior monitoring and emergency event detection	IVR-50	Deep learning sound recognition
[110]		2015	energy-related activity recognition in buildings	Smartphone	Locality-constrained linear coding method
[111]	Illuminance	2016	Occupant comfort, energy consumption	Light sensor, headlight, PIR sensor	Statistic analysis
[112]		2015	Lighting control for comfort and energy efficiency	Light sensor, headlight, PIR sensor	Statistic analysis
[113]		2018	personalized visual satisfaction	Illuminance sensor, HDR camera	Bayesian modeling
[114]		2019	Occupant-centric lighting control	LightLearn hardware configuration (light sensor, RPI)	Reinforcement learning

Table 7. The list of references for building environment sensing.

No.	Type	Year	Research Object	Sensing Device	Data Processing Method
[101]	CO2	2019	Occupancy counting	CO2 sensor	Statistical analyses
[102]		2015	Occupancy counting, occupancy activity	CO2 sensor, beam-break sensor	Statistical analyses
[103]		2019	Occupancy counting	CO2 sensor	stochastic differential equations
[104]	IAQ	2017	Influence of occupant behavior on energy	IEQ monitors	Model-based simulation, statistical modeling
[105]		2019	Occupant behavior	Customized smart sensor node	Agent-based modeling
[106]		2019	Influence of IEQ on occupant behavior	SHT30, T6703 CO2 Module, SP3S-AQ2, MEVIU	weighed Euclidean distance
[107]	Sound	2020	Occupant behavior monitoring and emergency event detection	IVR-50	Deep learning sound recognition
[110]		2015	energy-related activity recognition in buildings	Smartphone	Locality-constrained linear coding method
[111]	Illuminance	2016	Occupant comfort, energy consumption	Light sensor, headlight, PIR sensor	Statistic analysis
[112]		2015	Lighting control for comfort and energy efficiency	Light sensor, headlight, PIR sensor	Statistic analysis
[113]		2018	personalized visual satisfaction	Illuminance sensor, HDR camera	Bayesian modeling
[114]		2019	Occupant-centric lighting control	LightLearn hardware configuration (light sensor, RPI)	Reinforcement learning

3.5. The Sensing of Building Consumption

3.5.1. Smart Meter

Smart meters and smart plugs are the most prevalent methods for sensing building consumption. The smart meter is an electronic device that records the energy or water usage of a building in real-time and facilitates bi-directional information exchange between the demand side and utility suppliers [115]. The key component of a smart meter is the microcontroller unit (MCU), which has powerful data storage, analysis, and intelligent decision-making capabilities. If connected to a wireless network, the smart meter may also monitor and regulate smart home equipment according to the directions of the occupant [116]. In the newer generation of smart meters, the human–machine interaction (HMI) technique has been used to improve the information feedback experience. The smart meter can interact with users through multimodal information, such as the visualization or the voice broadcast of energy consumption results, in the hopes of encouraging end-users to practice energy-saving behaviors [117]. In addition, because smart meters can provide residents with feedback about their power consumption, they have been used to develop behavioral intervention systems to encourage residents to adopt energy-saving practices in their daily activities [118,119].

The smart meter has benefited the HBI research through its capabilities of real-time monitoring of power consumption and big data provision since the consumption data has been utilized as an effective data source to infer occupancy status or energy consumption patterns. Utilizing machine learning techniques is a useful method for modeling the complicated interactions between occupants and buildings [120,121]. Based on the smart meter data, Singh et al. [122] used a frequent pattern mining method to analyze variations of occupant behaviors in households, including differences in energy consumption, appliance usage, and time. In order to fulfill the goal of remote control, the study team extracted features from smart meter data analytics during the building commissioning phase, including the usage type, performance class, and operation group.

3.5.2. Smart Plug

The smart plug is an intelligent power socket or converter with remote-control power on and off capabilities that can collect the electricity load information of the domestic or office equipment in real time, transmit the consumption data to the household energy hub, and provide feedback to the power user via a mobile application. In HBI research, the state of electrical appliance usage is considered an indicator of occupancy. For example, in an office environment where practically everyone uses one computer, the utilization situation of the computer can be considered an occupancy indicator, and the electricity consumed by the computer can also be correlated with the office’s occupancy [123]. Based on this consideration, the room occupancy patterns in an office building were identified by reading the power consumption data of office computers with a smart load meter. The authors also stated that, compared to the CO₂ sensor, the power consumption data could provide more accurate results of occupancy patterns and occupancy numbers [123].

Similar to the smart meter, the smart plug has a consumption feedback function for energy-saving behavior intervention. In a behavioral intervention experiment conducted by Jenkins et al. [124], the research team selected 46 offices and upgraded their existing electric outlets to smart plugs for three purposes: collecting energy data from each outlet, the remote control of outlet, and transmitting the energy data to web and mobile application. As a result, the smart plugs successfully engaged workplace occupiers to cut plug load electricity consumption by 32%, according to the article. The references reviewed for the sensing of building consumption are listed in

Table 8. The list of references for building consumption sensing.

No.	Type	Year	Research Object	Sensing Device	Data Processing Method
[118]	Smart meter	2018	Energy consumption	HEMS	Statistic analysis
[119]		2019	Energy saving	Smart meter, in-home display	Statistical analysis
[121]		2017	Occupancy detection	Smart meter	Monte Carlo simulations, RF
[122]		2021	Building usage type, operation pattern	Smart meter	Machine learning-RF
[123]	Smart plug	2017	Room occupancy pattern	Smart plug, CO2 sensor	Statistic analysis
[124]		2019	Occupant engagement	Smart plug	Statistic analysis

.

3.6. The Fusion of Multi-Sensing System

Table 9. The list of references for the fusion of multi-sensing system.

No.	Type	Year	Research Object	Sensing Device	Data Processing Method
[125]	RGB PIR Magnetic reed contact IAQ Sound illuminance smart plug	2016	Occupancy counting	Video-camera, Motion detector, Window contact, Smart plug, Microphone, IAQ monitor	Decision tree algorithm
[126,127]	Wi-Fi, CO2, Temperature, humidity sensor, RGB	2018 2019	Occupancy prediction	Wi-Fi probe, Web camera, IAQ monitor	ANN, KNN, SVM
[128]	Environmental, Audio, image	2022	Occupancy detection	IAQ monitor, Video Camera,	Occ-STPN, RF, Few-shot model

presents four examples for the employment of multi-sensing fusion system in HBI research. Due to the fact that each type of sensing technology has its own applicability and limitation, a single sensing technique rarely provides sufficient reliable information; thus, a number of studies have adopted the strategy of multi-source data fusion to reduce uncertainty with the data and improve the quality of the information extracted. Aiming to contribute to the occupancy estimation, a sensor network was deployed by Amayri et al. [125] in an office for data acquisition, which consisted of two video cameras, a luminance sensor, IAQ sensors, a power consumption sensor, reed contact, acoustic sensor and PIR sensor (Figure 10), and the authors used decision tree algorithm for data processing.

In order to increase the accuracy and reliability of the occupancy prediction model, Wang et al. [126] utilized environmental sensors (air temperature, relative humidity, CO₂ concentration), Wi-Fi probes, and video cameras to integrate a fused sensing network for obtaining occupancy data. The research team set up their on-site experiment in a 200 m² graduate student office with 25 residents and conducted two experiments (Figure 11). In this article, they compared three machine learning algorithms, k-nearest neighbors (KNN), support vector machine (SVM), and artificial neural network (ANN). They reported that the ANN-based model with a fusion data source appeared to have the best performance, while the SVM model was more suitable with Wi-Fi data. The fused data outperformed the independent sensor data in terms of model accuracy and robustness for occupancy prediction. In article [127], the authors developed an adaptive lasso model to identify the features with high correlation to the actual occupancy profiles and stated that indicators of CO₂ concentration, temperature, and Wi-Fi signal were the most correlative.

In Ref. [128], the authors developed a high-performance occupancy detection system incorporating sensor data of various modalities, including time series environmental data (temperature, humidity, and illuminance), image data, and acoustic data. In order to achieve the highest prediction performance across different types of sensing data, as shown in Figure 12, the authors developed a multimodal sensor fusion framework built with different machine learning models to understand the occupancy patterns revealed in the physical data streams. After developing the inferencing model for each modality of sensing data, the research executed a combination of weighted probability and knowledge-based decision fusion in the final stage of this framework and multiple public datasets were utilized for validation.

Aiming to determine the impact of sensor placement on the data accuracy of a fusion of sensing systems, Azizi et al. [129] conducted a study with 18 multi-sensor devices installed in three workplaces. Multiple sensors for detecting PIR, temperature, carbon dioxide, humidity, and light were put in each device. Their findings indicated that the location of PIR and CO₂ sensors had a significant impact on the accuracy of occupancy detection. Placing sensors in the middle of the office ceiling may not provide very accurate data, whereas placing sensors under office desks could increase the accuracy of presence detection by up to 84%. In addition, the sensor’s location has a significant impact on the measurement of environmental indicators, particularly temperature and lighting data.

With the penetration of smart technology in a household, the smart home has become a data-producing source, as the various sensors embedded in the smart products are continuously recording the household’s daily data, such as the indoor air quality, energy consumption, occupancy, etc. Therefore, if the data can be appropriately used, some novel services that improve the lives of occupants would be incubated. For example, Varlamis et al. [130] developed an energy-saving application–an online recommendation system based on the fusion data of various sensors, user behavioral habits, and feedback from occupants. This application could provide personalized energy-saving suggestions to residents at the appropriate time, as well as record user feedback and refine future recommendations.

4. Discussion

4.1. The Facts behind the Data

The sensing technologies enable the research to acquire more comprehensive and extensive data to study the occupants, the building, and their interrelationships. However, it should be noted that data is only intended to provide more evidence to validate the proposed hypotheses or models, and scholars should attempt to uncover the facts that are hidden behind the data. For instance, Wi-Fi connection data has been validated as an effective method for determining occupancy numbers, provided that occupants have a strong intention to connect to Wi-Fi in the building. However, because occupants have multiple devices that require an internet connection, the occupancy counting derived from connection data may be higher than the actual number.

The feedback and experience of occupants are essential issues of HBI research. Although a number of studies have adopted wearable sensors to measure human physiological data in order to provide more objective and scientific material for scholars to understand the physical sensations of building occupants, the subjective bias-prone questionnaire survey will always be indispensable for result validation, as a prevalent method to investigate occupant’s perception and experience. In Ref. [61], after analyzing body temperature and EEG data, the authors also used a questionnaire to collect subjective views of the building environment for cross-validation and discovered a strong correlation between EEG data and subjective questionnaire, and in Ref. [75], in order to thoroughly evaluate the comfort perception of occupants, the indoor air quality sensors, the wearable physiological sensors, and the subjective survey were all used to collect data simultaneously. Except for the built environment, the physical sensation of occupants will also be influenced by behavioral and phycological factors, which is an important research area in HBI [131].

4.2. The Cost-Effectiveness of the Fusion of Multi-Sensing System

Increasing numbers of research utilize multi-type sensors to collect data from multiple sources in order to accomplish comprehensive sensing of the human-building-environment system. Theoretically, the approach of data fusion might compensate for the flaws of a single sensor, offer cross-reference between each sensor, and improve the accuracy and reliability of the research [126]; however, it also raises a number of concerns. The first concern is the cost-effectiveness relating to the sensor and advanced data processing techniques. In Ref. [127], the authors predicted occupancy using a fusion of multi-sensors, but they emphasized that their study did not consider the trade-off between accuracy and sensor cost. In Ref. [128], the authors also indicated that they would consider low-power sensors and low-cost computation devices in future work. Therefore, the choices of the sensors and computational infrastructure in the study of multi-sensing fusion should be carefully considered.

Secondly, the growth of the data volume and data type will lead to more sophisticated data synchronization and processing. The machine learning (ML) technique can derive distinctive features from unprocessed sensor data collected from the building operation phase to obtain comprehensive knowledge about occupant behavior. Various ML models have been utilized in previous studies, such as the ANN model [126,132], KNN model [11,133], SVM model [11,33,133], RF model [11,98,122], etc., because the performance of different models can vary greatly, as Wang et al. [126] demonstrated that after comparing the analysis results of the fusion data, only the ANN model improved its performance, while the KNN model and SVM model did not show the difference from the individual data set. However, in the occupancy detection study of Yoon et al. [11], the authors reported that the RF model proved to be the highest accuracy compared to the KNN model and SVM model. Therefore, the selection of the ML models should be handled with care, taking into account the volume of the data set, as well as the features of the data, like type, structure, and format, etc.

4.3. The Privacy Issues Involved with Data Acquisition

As the goal of data collection in HBI research is to monitor, detect, or infer the interactions between occupants and buildings, the privacy issue associated with the data-sensing process is of the utmost importance. In the majority of the reviewed publications, human-in-the-loop data collection methodology was used to collect data, indicating that participants were highly involved in the research process. For example, in Ref. [50], in addition to the sensors used to record the occupant behavior data, a depth camera was used for manual observation as ground truth proof. In Ref. [125] and [127], test beds equipped with a variety of sensors were developed to observe occupant behavior over a length of time without interruption. In these situations, it must be a research ethic to preserve the privacy of experiment participants. The residents’ reluctance to embrace smart meters is also hindered by the possibility of privacy leakages [134]. According to Refs. [122,135], smart meter data mining techniques can reveal an occupant’s daily schedule and living habits; therefore, smart meter data must be protected carefully lest the data be used by criminals. Some technical approaches have been proposed for privacy protection in smart environments, which could serve as a reference for future research [136].

5. Conclusions

This paper presents a systematic review of the application of sensing technology in the research of HBI. According to the research contents of HBI, the sensing technologies were organized into six application scenarios: occupancy status, the physiological indicators of occupants, the conditions of building components, the attributes of indoor/outdoor environment, the consumptions of the building, and the fusion of multi-sensing system.

Initially, the research of occupancy status includes the occupancy counting, occupancy presence, occupancy location, and occupancy trajectory, and four types of prevalent sensing technologies were reviewed: camera-based sensing, which includes the RGB camera and depth camera; infrared-based sensing, which includes the active infrared, such as the break-beam sensor, and the passive infrared, such as the PIR sensor; the radial frequency signal based sensing, which includes Wi-Fi; and the ultrasonic sensor. Secondly, the physiological indicators of occupants are an effective method for determining the comfort level of occupants in a building. These indicators can be acquired through a variety of biosensing techniques, such as those related to the human brain (EEG, ERPs, fMRI), muscle and skin (EMG, EDA), and cardiac function (ECG) monitoring. In addition, as a result of the development of wearable sensors, the wristband with multiple health monitoring functions has gained popularity in relevant research, which was also examined. Thirdly, by detecting the changes in building components such as windows, doors, and floors, data can be collected to reveal and predict interactions between persons and buildings; the magnetic reed sensor and the vibration sensor are generally utilized for this purpose. Fourthly, the article discussed a series of environmental sensors, including the indoor air quality sensor, the acoustic sensor, and the illuminance sensor, which could acquire environmental indicators to enable the study to examine the reciprocal interaction of the building environment and residents. Fifthly, the sensing technologies incorporated within the smart meter or smart plug were reviewed in terms of their functionalities of real-time consumption monitoring and data feedback, allowing HBI studies to infer the occupancy status or occupant behavioral pattern using machine learning algorithms. Lastly, a number of studies have leveraged the fusion of multi-sensor systems to perform full sensing of the human-building-environment system to compensate for the limitations inherent to a single sensor and improve the research’s reliability and accuracy.

Additionally, the results of the review were discussed from three perspectives. Firstly, thanks to the sensing technology, the researchers were able to collect more data than ever before to support their research; yet, the data could not show all, and certain realities concealed by the data deserved in-depth investigation. Moreover, in the application of multi-sensor fusion, the cost-effectiveness of advanced data processing techniques should be considered. The increase in data volume will raise the difficulty of data synchronization and data analysis. The machine learning technique is a primary method to solve big data analysis; however, because the performance of different ML models varies greatly, it is advised that researchers select the appropriate ML model with caution. Lastly, the issues of personal privacy caused by the expansion of sensing technologies are so important that the research staff should give them adequate attention.