1. Introduction and Literature Review
A significant proportion of global energy demand and emissions is due to the built environment sector [
1]. Taking into account the total life cycle of the building, the energy demand of buildings is up to 35% of the total final energy consumption, and this is growing fast [
2,
3]. Hence it is crucial to minimise the building energy usage in order to meet the global carbon emission reduction target. A significant proportion (40%) of the operational energy use is due to the use of HVAC [
4]. This is even higher in areas with very hot or cold climates [
5]. Minimising the consumption and enhancing the efficiency of HVAC will go a long way towards the development of a low carbon economy and future. However, the comfort and well-being of the occupants should also be considered when developing solutions [
6]. Solutions such as demand-driven controls can achieve substantial energy savings by reducing or eliminating avoidable energy usage and provide a comfortable indoor environment for occupants [
7,
8].
Occupancy behaviour, activities and patterns are significant factors affecting the utilisation of HVAC [
9]. For example, rooms in buildings are not fully utilised or occupied during the day, and in some cases, some rooms are routinely unoccupied. While the HVAC systems use conventional control system and operate by using a fixed set point schedules which assume max occupancy during the entire working week. The use of fixed set points in combination with varying occupancy activities and patterns could lead to rooms frequently being over- or under-conditioned, which may lead to significant energy wastage [
10,
11]. The studies [
12,
13] collected occupancy data from various buildings and have shown that those average daily occupancy rates were rarely over 60%, particularly in single-person offices. While equipment or appliances in offices can be kept in operations during the entire working day, irrespective of the patterns of occupancy [
14]. This also contributes to the disparity between the predicted and actual energy usage or the energy performance gap.
Hence, the use of solutions, such as demand-driven controls or occupancy-based controls is necessary. Such solutions can adapt to occupancy patterns in real-time and optimise HVAC operations while providing comfortable conditions [
15]. These systems reduce energy consumption by optimising the scheduling of the HVAC as well as other building systems by using the occupancy information [
16,
17]. Energy savings can be achieved by the demand-driven solutions by adjusting the setpoints to reduce the temperature difference between the outdoor and air-conditioned indoor space and reducing the operation time of the systems [
18,
19,
20].
In order to effectively develop and implement demand-driven control strategies for HVAC, accurate and real-time information on real-time occupancy patterns is necessary [
21]. The occupancy information can be collected in real-time using sensors and monitoring technologies [
22], including passive infrared sensor (PIR) or motion detectors [
23], environmental sensors [
24], wearable sensors [
25] and Bluetooth or Wi-Fi sensors [
26]. The capabilities of such strategies have been shown in previous works [
21,
22,
23,
24,
25,
26], which focused on detecting the number and positioning of the occupants in a space. However, research on detecting the occupancy activities and predicting heat gains which can affect the indoor environment conditions are limited [
27,
28]. The activities of occupants can affect the internal heat gains (sensible and latent heat) in spaces directly [
27] and indirectly [
28]. For example, a person walking around the space will have a different heat emission or heat gain as compared to a person sitting. In addition, the usage of equipment in offices such as desktops and printers can also have an impact on the internal heat gains. The information on the heat emitted by the occupants performing different activities and usage of equipment can be utilised to better assess the actual requirements of spaces in terms of heating, cooling and ventilation [
29]. A potential solution is to use artificial intelligence (AI)-based techniques such as deep learning and computer vision to accurately detect, recognise and predict these information in real-time [
30].
Deep learning is a machine learning technique which has been utilised to implement tasks such as classifying objects, recognising speech, detection of pedestrians with high accuracy [
31]. Additionally, many studies have shown that the convolutional neural network (CNN) can perform well in computer vision tasks [
32]; hence CNN was selected as the algorithm to enable real-time detection and recognition in this study. It is widely used as it can directly feed the original image into the model instead of performing the complex pre-processing of the image [
33].
Deep learning and computer vision methods have recently been adopted in the built environment to enhance building system operations. Zou et al. [
34] proposed a deep learning-based human activity recognition scheme to automatically identify common activities in offices which assessed a 97.6% activity recognition accuracy. Markovic et al. [
35] used a deep feed-forward neural network to model the opening of windows in offices which showed an evaluation accuracy between 86% and 89%. These studies indicate that deep learning and computer vision methods have large potential to accurately evaluate the energy behaviour in the buildings and further optimise the operations of building management systems.
To form effective detection models, a suitable deep learning framework platform is required. Many deep learning framework libraries and platforms such as TensorFlow, PyTorch and Keras are highly popular and are recommended according to Google Trends (as of February 2020) [
36]. Along with the comparison of deep learning frameworks by Fonnegra et al. [
37], it suggests that TensorFlow is one of the most employed tools used for deep learning because of its capabilities, compatibility, speed and the support it provides. TensorFlow [
38] allows the testing of configurations of deep learning algorithms and demonstration of their robustness. According to previous works, many choose TensorFlow as the desired platform for the development of solutions for building-related applications. This includes [
35] where TensorFlow has been used as a platform to train the desired deep learning model. Vázquez-Canteli et al. [
39] fused TensorFlow technique with building energy simulation (BES) to develop an intelligent energy management system for smart cities and Jo and Yoon [
40] indicated that TensorFlow was used to establish a smart home energy efficiency model. Additionally, the provision of pre-existing open-source deep learning-based models by TensorFlow, such as the CNN TensorFlow object detection API [
41] enabled researchers to use this framework as the base configuration for detection-based applications. This includes the applications by [
33,
42,
43], which effectively fine-tuned the model to improve accuracy and to adapt for the research desired detection purposes. Therefore, the TensorFlow platform with the CNN object detection API was employed for the development of a suitable model for this study.
1.1. Literature Gap and Novelty
Previous works [
34,
44] have shown the capabilities of computer vision and deep learning methods to detect and classify human presence and movement. Many of the studies focused on improving the performance of such approaches such as detection accuracy, robustness, speed etc. However, studies focusing on using the provided data to seek solutions for minimising the unnecessary building energy usage have been limited. In addition, no work has employed computer vision and deep learning methods to predict the heat emitted (sensible and latent) by both occupants and equipment in a building space, which can affect the temperature and moisture levels and subsequently the operation of the HVAC. Furthermore, studies that conducted field testing of computer vision occupancy detection approaches in office spaces have been limited.
1.2. Aims and Objectives
To address the literature gaps, this study aims to detect and recognise the real-time usage of multiple equipments and occupancy patterns in a room or space using a computer vision and deep learning approach. A Faster R-CNN is developed and trained for classification and deployed to a camera for detecting the occupancy activities and equipment usage. This method identifies the multiple occupants and equipment within an indoor space and the activities performed by each of them. The model performance is evaluated in terms of the different evaluation metrics. Field testing in an actual office space at the University of Nottingham is carried out to validate the proposed approach and assess its capabilities. In order to evaluate the impact on the cooling and heating energy demand, the case study building was modelled and simulated in a building energy simulation (BES) tool and the data generated using the proposed approach were used as input. A comparison between the heat emission profiles of the proposed approach (also called here deep learning influenced profile or DLIP) and fixed or static profiles is conducted.
4. Conclusions and Future Works
This study proposes a vision-based occupancy and equipment usage detection approach for demand-driven control systems to minimise the unnecessary energy usage and enhance thermal comfort. The method can detect and recognise multiple occupants’ and their activities and equipment usage within building spaces in real-time. A Faster R-CNN was developed, trained and deployed to a camera for real-time detection of occupancy activities and equipment usage. The proposed models were validated through the use of a set of testing data, and the results suggested the equipment detection model achieved an accuracy of 80% with an F1 Score of 0.8889 and the occupancy model achieved an accuracy of 97.09% with an F1 Score of 0.9270. It indicated that the model is suitable for live occupancy and equipment detection.
Experiments were conducted within an office room to assess the capabilities and accuracy of the proposed approach. It showed that overall occupant’s activities and equipment detection accuracies of 97.32% and 80.80% were achieved respectively, which are in accordance with the test accuracies. It presents the capabilities of the model to detect the equipment usage and recognise the differences between the corresponding human poses for each specific activity. Four test scenarios in terms of the use of profiles were set up to investigate the impacts of the application of the deep learning vision-based detection method towards building energy performances. Using BES, the building was modelled and simulated using a static or fixed profile and DLIP profile to predict the influence on energy consumption. The results showed that the internal heat gains achieved by typical profiles were much greater than the gains achieved using the deep learning approach. It also suggested that when both equipment and occupancy deep learning methods were used (Scenario 4), a 65.75% and 32.74% decrease of equipment and occupancy gains could be achieved respectively in comparison to Scenario 1 results.
This highlights the benefits of using the deep learning approach to provide deep learning-based profiles to HVAC control systems to enable the achievement of meeting real-time requirements for reducing the unnecessary building loads.
Overall, the proposed approach presents the capabilities of detection and recognition of multiple occupants’ activities and equipment usage and provides an efficient alternative to accurately detect and estimate the internal heat gains. To further optimise the detection performance, some improvements are required to be applied to the approach to decrease the error rate as the error rate could result in an inaccurate prediction of the energy demand and further causes an uncomfortable indoor environment for occupants. In addition, a strategy, which transfers the data clustered from deep learning model to occupancy and equipment profiles and feeds them into BEMS to automatically adjust HVAC setpoints, is required to be developed.