Deep Neural Network Based Ambient Airflow Control through Spatial Learning

Abstract

As global energy regulations are strengthened, improving energy efficiency while maintaining performance of electronic appliances is becoming more important. Especially in air conditioning, energy efficiency can be maximized by adaptively controlling the airflow based on detected human locations; however, several limitations such as detection areas, the installation environment, and sensor quantity and real-time performance which come from the constraints in the embedded system make it a challenging problem. In this study, by using a low resolution cost effective vision sensor, the environmental information of living spaces and the real-time locations of humans are learned through a deep learning algorithm to identify the living area from the entire indoor space. Based on this information, we improve the performance and the energy efficiency of air conditioner by smartly controlling the airflow on the identified living area. In experiments, our deep learning based spatial classification algorithm shows error less than ±$5^{°}$. In addition, the target temperature can be reached 19.8% faster and the power consumption can be saved up to 20.5% by the time the target temperature is achieved.

1. Introduction

With the development of information and communication technologies such as the computer, cloud, and Internet of Things (IoT), the machine learning technologies for image processing and voice processing have been combined with the deep learning technology and their application fields are spreading throughout the industry [1,2]. Moreover, as energy efficiency regulations are strengthened, increasing the energy efficiency of electronic appliances used in daily life becomes an important task [3,4]. Among electronic appliances in the living space, air conditioners use the most energy, and because of that numerous techniques have been studied to maximize energy efficiency while maintaining the same performance [3,5]. To address these problems, various human body detection sensors are used to detect the presence of humans and control the airflow of the air conditioner [4,6]. However, there are several technical problems with detecting a person and control an airflow based on the limited viewing angle of the sensor, with the constrained condition of the installed position of the product and the number of sensors [3,7]. In addition, when the vanes for discharging cold air are fixed, the cold air is concentrated only at a specific region, and cause an uneven temperature variation in the indoor space. Furthermore, it also causes a feeling of discomfort when the cool air directly affects the people in the living area [3,5]. To solve this problem, the entire wind swing should be controlled so that it can make the room uniformly comfortable. However, even in the case of a full swing, the extent of swing range is a fixed for the product, and it cannot be adaptively controlled for the cooling of the room based on the installed environment [3,4,6,7]. Consequently, cooling air is sent to walls and windows, leading to the loss of cooling, movement of curtains, and frosting of windows.

In this study, to maximize the energy efficiency while maintaining the cooling performance of the indoor air conditioner, we minimized the loss of cold air by controlling the airflow differently with the condition of the living and non-living areas of the indoor space [3,4,7,8]. In addition, in the space where the air conditioner is installed, the location of a person in the room is recognized in real time to ensure that the cooling air is majorly directed around the living area to reach target temperature more quickly [9,10,11,12]. The challenges of real-time tracking of the human location and the assessment of the installation environment of the air conditioner are simultaneously addressed by learning the indoor space environment and human location through a deep learning algorithm. Our approach uses a vision sensor for human detection and subsequently controls air conditioner according to the living and non-living areas [1,9,10,11,12]. To classify the living and non-living areas, a spatial learning algorithm using deep learning on detecting the user’s movement and changes in the indoor environment was used. In addition, our human body detection algorithm can operate effectively even on an embedded system with very low computing power with a cheap and a low resolution camera mounted on an air conditioner.

2. Problem Definition

In this paper, majorly three problems are addressed to optimally control the air conditioner for maximum energy efficiency and room occupant’s comfort.

The first problem is the detection of a resident in the room. For an air conditioner which generally installed at home, the computational performance of the embedded hardware is so limited. So, any general detection algorithms requiring high computational performance cannot be adopted. Therefore, the main challenge of the problem is achieving good detection performance with low computational resources.

The second problem is the estimation of frequently staying area of a resident based on the detected locations. When estimating resident’s main living area (based on human detection results), the major challenge is devising an algorithm to accurately classify it based on accumulated human body detection results.

Finally, based on the estimated main living area, provides an optimal control of air conditioner for indoor users. There are various control options for air conditioners, such as wind direction or power consumption, thus, major goal is selecting the optimal control method while considering user comfort and energy efficiency.

3. Related Work

Nowadays, there is active research on the field of human body detection, indoor localization and their application for smart home appliances [13,14]. For human body detection, various sensors such as passive infrared (PIR), thermopile, and vision sensors are adopted in the home appliances. A PIR sensor has a low cost for both price and computation, therefore, it can be easily used for real-time detection; however, it has too short detection range and it is hard to derive the distance and the direction from the sensor output [3,7]. For this issue, multiple sensors can be used, but, such approach is not easily applicable to real products because of the limited number of mounting positions in the product and the increasing cost. Human body detection based on a thermopile sensor detects the temperature differences between the skin and the ambient and has a wider detection range than PIR. However, for detection of objects in long distance, a multi-zone thermophile is required [3,7] which is expensive. Camera vision sensors are becoming more popular in the home appliances owing to the proliferation of smart devices and high-speed micro-controller units [7,15]. Because of its low cost and real-time performance, it is widely adopted in various application fields with object recognition and tracking. In human detection problem, camera sensor is used in several algorithms such as upper body detection algorithm which is based on the local binary pattern (BP) and histogram of oriented gradient (HOG) [16] or head-shoulder (upper body) pattern detection. There is also a detection method which utilizing vertical symmetry or general symmetry of the target with camera sensor. Generalized symmetry transform (GST) [17] detects symmetry based on gradient information is an example of general symmetry detection based on camera [18]. The HOG feature-based human body detectors shows good performance and widely used in the field of pedestrian detection [19]. However, they are vulnerable to lighting and environment changes.

In this research, we propose an efficient air conditioner control method based on the indoor user localization which is obtained from human body detection with spatial learning [20,21,22,23]. The main contribution of this work is a new method to implement an intelligent air conditioning for energy efficiency, which monitors human activity in a living room and adaptively controls airflow based on human locations. To achieve optimal performance, we propose a cost effective novel encoding protrusion map for representing human activities as well as omega human body detector using a low-spec camera, and deep learning algorithm successfully generates living areas which are important to control airflow. Theoretical contribution mainly corresponds to the cost effective pre-processing including protrusion map for deep learning. From given airflow control options such as direct wind control (which directs the airflow to people), intermittent wind control (which directs airflow to areas without people), and on/off control of air conditioners, the proposed method chooses an optimal option so that it can ensure faster cooling and reduction of the unnecessary energy consumption [3,6].

4. Technical Approach

The proposed spatial learning algorithm acquires the occupant’s positional information through a human body detection sensor attached to the air conditioner and frequently estimates occupied area in the living space [24,25]. As shown in Figure 1, when the area detection function is performed from the indoor unit, the human body detection results are accumulated and the accumulated data are input to the machine learning algorithm to distinguish the majorly living area and the unused area. The estimated living area through human body detection in the living room of a real home are as shown in Figure 2.

4.1. Human Body Detection

The proposed algorithm detects occupants who exist within 5 m of the camera of Quad-VGA (1280 × 1024) resolution and collect positional information. It majorly focuses on the simplicity and efficiency of a cost effective embedded system. It first detects moving objects from obtained video frames and finds human bodies through an Omega human body detection model. For robustness, it also verifies the human detection results by symmetry.

4.1.1. Moving Object Detection (MOD)

The motion region is the part of the foreground region of the image that changes with time. Therefore, it can be easily found by extracting regions that change over time from successive image frames. If there is a background image obtained with no foreground objects, the moving object region can be extracted by subtracting the background image from the moving image frame. However, this approach is problematic because the lighting or surrounding objects in the background scene can also change with time. A stable background subtraction algorithm should be able to handle repetitive motion and long-term scene changes in the scene. For this purpose, an averaging method is adopted to obtain a stable background region. In this method, a plurality of images are averaged by Equation (1). The background image at time t is obtained as follows, where N is the number of images added for averaging.

$$\begin{array}{c}\hfill B(x,y,t)=\frac{1}{N}\sum _{i=1}^{n}V(x,y,t-i).\end{array}$$

(1)

This equation represents the mean of the pixels for given images. The N depends on the frames per second and the degree of motion in the image. After computing the background image $B(x,y,t)$, it is subtracted from the image $V(x,y,t)$ at time t to obtain the foreground region as in Equation (2).

$$\begin{array}{c}\hfill \left|V(x,y,t)-B(x,y,t)\right|>Th.\end{array}$$

(2)

Here, $Th$ is a threshold value. The threshold in Equation (2) is used for segmentation of the human body from background image, which is determined by a heuristic method. It can be considered an adaptive way of finding an optimal threshold. But, we just simply find the threshold for a living room with 300 ∼ 900 lux, which is usual illumination at a living room in daylight or night with fluorescent lamp. The foreground region extracted by Equation (2) is shown in Figure 3.

4.1.2. Omega Human Body Detection Model

Embedded systems have a limited memory and computing power compared to other conventional computing systems. This property places a limit on the performance of user recognition or detection algorithms in real-time applications. In this study, in order to implement a smooth and fast real-time detection and recognition system, we focused on the upper half of human body (which has $\Omega$ shape) for human detection as shown in Figure 4. Generally, the major problem in the human detection system is its robustness to diverse changes in the input condition, such as changes in body posture, shape, clothing and lighting. In other words, the main task for solving the detection problem is to find a unique feature that can characterize humans with robustness to diverse condition changes. From this perspective, the upper body of a person can be regarded as a unique feature which consistently keeps the shape of $\Omega$ in any posture or environmental variations. Therefore, it is used as the main feature for fast searching and robust human detection.

4.1.3. Human Body Detection Verification Through Symmetry

Most of the head-to-shoulder (upper body) patterns have a similar characteristic, whereby the left half and right half show vertical symmetry even though there are some variations on the pose and the views. As shown in Figure 5, the upper body of the human has symmetrical distribution.

For more general symmetry detection, a Generalized Symmetry Transform (GST) can be applied [17] which detects symmetrical components based on gradient information in the image [18]. Although it can be used even when there is no information on the symmetry axis and also good detection results can be obtained, it is not suitable for embedded systems because of its long computation time. In this research, the degree of similarity is measured by comparing the intensity histograms between the detected window areas. For symmetrical components detection, both upper-half body detection results and MOD results are used. The vertical symmetry axis position is computed based on the result of MOD and verified with the similarity of the intensity histogram between the left half and right half region, and also compared with the detection result of the upper half of the body. The human detection algorithm is implemented and executed on the embedded system (Dual Core CPU / RAM 2GB), and it takes from 3 to 4 s to detect a person based on the experiment average.

4.1.4. Distance Estimation for Detected Human Body

The position of the person in the room is two-dimensional information with axes of angle and distance. In our approach, the wind direction of the air conditioner is controlled based on the angle information, and the wind speed is controlled based on the distance information. Unlike the wind direction control of the air conditioner, the wind speed is relatively easy to control with the conditions of "strong" and "weak" according to the distance. For the distance measure, the average size of a person is used as a reference value, and the relative distance is assigned with respect to the reference size from the first to sixth step.

4.1.5. Exception Handling in Human Body Detection

For the exceptional case, such as a mirrored image of person on the TV screen, a threshold value is applied to the hue values of the detected region because the hue value of the human skin is different from that of the TV mirrored image. By this method, our system can detect only a real person without falsely detecting the mirrored image on the television.

4.2. Estimation of Indoor Spatial Information for User

The purpose of this algorithm is to estimate the main living area using a deep neural network (DNN) trained by human detection data. The detection data consist of the angle and distance from the installed camera.

4.2.1. Dataset Construction

For data collection, the surrounding environment is divided into 21 segments and each segment is further divided to 5 sub segments. A histogram is created by counting the number of human appearances in each segment. As shown in Figure 6, the area labels of task (1) and non-task (0) is assigned according to the frequency of human appearance.

The distance from the camera to the human body is recorded in 6 levels (0 to 5). The angle and distance information are used to construct a two-dimensional map of human body detection as shown in Figure 7. X and Y axes are the angle of living zone and the distance from the camera installed in the air conditioner to a resident, respectively. By using angle and distance information, the histograms represent a 3-dimensional distribution of living area of the residents.

In each indoor environment, input and target output pairs are configured for every 500 examples from 10,000 human detection results. Direct accumulation of 500 human detection histograms usually allows the accumulated detection results to be represented as points as shown in Figure 8.

Therefore, when a human body is detected as a single point, the top, bottom, left and right areas of a detected point are also cumulatively marked together to represent local protruding area as shown in Figure 9. In this case, even though the human body is detected only at the boundary point of specific region, the frequency of appearance in that local region can be increased at the same time. Therefore, the protrusion map can effectively represent spatial information.

$$\begin{array}{c}\hfill h_{ij}=\frac{2}{3}max\left(h_{ij}\right)\mathrm{if}{\left(h\right)}_{ij}>max\left(h_{ij}\right),\end{array}$$

(3)

$$\begin{array}{c}\hfill h_{ij}=0\mathrm{if}{\left(h\right)}_{ij}<\frac{1}{2}\overline{h}.\end{array}$$

(4)

The obtained protrusion map is further processed for the stable learning of DNN. To make the human detection values discrete, data are normalized and unresolved maximum and trivial minimum values are eliminated. For the unprocessed maximum, all values greater than the protrusion map’s maximum are replaced with $\raisebox{1ex}{$2$}\!\left/ \!\raisebox{-1ex}{$3$}\right.$ of the maximum. For trivial minimum, all values less than $\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.$ of the average value of the protrusion map are replaced with zeros. An upper threshold and a lower threshold is applied to distinguish each salient region in the histogram according to Equations (3) and (4). When we use the histogram directly, the results are discontinuous and also it may include noise information. In order to remove the noise, we simply use the Equation (4) to remove the histogram result with less than the average histogram value. Also, when we build the protrusion map, we use stacked crosswise histogram. Since it may over count for the boundary information for the final histogram, we normalize the maximum protrusion map as the $\raisebox{1ex}{$2$}\!\left/ \!\raisebox{-1ex}{$3$}\right.$ histogram max value as shown in Equation (3). Those thresholds such as half of average and $\raisebox{1ex}{$2$}\!\left/ \!\raisebox{-1ex}{$3$}\right.$ histogram max are chosen by a heuristic method through many trials and errors. The protrusion map after data processing is completed is shown in Figure 10.

It also defines the swing area for swing operation (wind direction control). For the estimation of the swing area, the second largest area is found while excluding the largest and largest mask areas and linear interpolation is performed based on 3 × 3 window with local summation.

4.2.2. Deep Neural Network based Spatial Learning

In spatial learning, every point in the protrusion map should be classified as either the living areas or non-living areas by DNN. Since DNN needs input and target pairs, we use the protrusion map for training, which is constructed by accumulating human body detection results.

In the pre-processed protrusion map, the gray and black areas are labeled as living area (1) and the white areas are non-living area (0). Figure 11 shows the proposed DNN architecture.

Figure 12 is an example of the pre-processed protrusion map directly used for DNN training. For each indoor environment, DNN prediction results are accumulated for every 500 detection from 10,000 human detection data (for the estimation of temporal change in the living area of residents). When accumulating previous and current predictions, following time weighted function is applied.

$$\begin{array}{c}\hfill y_t=\lambda {\widehat{y}}_t+(1-\lambda ){\widehat{y}}_{t-1}.\end{array}$$

(5)

In Equation (5), $y_t$ represents the living area at time t and ${\widehat{y}}_t$ is the predicted result of DNN at time t. $\lambda$ is time factor and we set it as 0.3 to emphasize the previous prediction of DNN.

4.3. Air Conditioning Control based on Spatial Information (Smart-Care)

A smart-care control logic is applied to efficiently transmit air based on the living, moving and unused area information obtained from a spatial learning algorithm and control direct and indirect air switching function. As shown in Figure 13, at the initial stage of cooling, the smart-care function rapidly achieves the target temperature, regardless of the set temperature (Rapid Mode) by exhibiting the unconditional maximum cooling capacity. After that, the target temperature is maintained by setting the air conditioner to a state without any direct air flow (Comfortable Mode). In rapid mode, since the air conditioner operates at the maximum power consumption, the power consumption increases rapidly as the time of the quick mode becomes longer. In the comfortable mode, it is difficult to consistently maintain the set temperature if there is a continuing loss of air flow to the area where people are not living.

The smart-care function based on the spatial learning algorithm adjusts the swing angle of the left and right vanes during the rapid mode or comfortable mode so that the airflow is provided only to the living area.

Table 1. Control of air conditioning operation with and without spatial learning (smart-care).

Rapid Cooling Mode
Spatial Learning	Left, Right Vane	Up, Down Vane	Fan Speed
No	Full Swing	Standard Angle	High
Yes	Living Area	Standard Angle	High
Comfortable Cooling Mode
Spatial Learning	Left, Right Vane	Up, Down Vane	Fan Speed
No	Full Swing	Upper	Low
Yes	Living Area	Upper	Low

lists the behavior of a smart-care function with spatial learning. When spatial learning is not applied, the left and right wind blades swing for the entire range that is initially set. When the spatial learning is applied, the air current is concentrated to the living area where people usually stay for a long time, during the rapid mode. In the comfortable mode, weak indirect wind is sent to the whole area except the wall and the window region where people do not stay. By reducing such loss of cooling air flow, the entire cooling can be performed more quickly in the rapid mode while saving the energy and also reduce weak cooling effect in the comfortable mode.

The smart-care learning system proposed in this study is operated by following process. In the module that performs the human body detection, the spatial learning algorithm is run to obtain the protrusion map to identify the living space. The obtained result is transmitted to the main controller of the indoor unit of the product, and the main controller transmits the information to the vane controller during the activation of the smart-care function to restrict the airflow only to the living area. In the future, we will further enhance the performance of our algorithms with collected big data.

5. Experimental Conditions and Results

5.1. Living Area Detection

To evaluate the performance of the living area detection with the deep learning based spatial learning algorithm, an air conditioner is fixed in the space as shown in Figure 14, and the time taken to identify the living area is measured while changing the area (a ∼ b).

After activating the space sensing function, the experimental target (approximately one to three people) moves freely within the specified range. In the experiment, if the spatial learning result obtained at a specified area has angle error less than or equal to ± 5°, it is regarded as the area sensing completion and the execution time is recorded. Considering the amount of data required for initial learning (350 data) and the average detection time (6 s), the detection experiment is considered to be successful when the time required for the space detection is measured to be less than 40 min.

Table 2. Results of Variation of Living Area experiment.

Experiment Conditions
Class	Update	Angle (°)	Detection Area (°)
Rotation Product (Continuous Leaning)	Non Living → Living	Default	0 ∼ 30
	Non Living → Living	5	0 ∼ 35
	Non Living → Living	15	0 ∼ 40
	Non Living → Living	30	0 ∼ 55
	Non Living → Living	60	0 ∼ 85
	Living → Non Living	100	90 ∼ 105
	Non Living → Living	75	65 ∼ 105
Experiment Results
Zone Detection (°)	Detection Error (°)		Test Time
0 ∼ 25	−5		34 m, 28 s
0 ∼ 35	0		7 m, 27 s
0 ∼ 35	−5		4 m, 1 s
0 ∼ 55	0		6 m, 10 s
0 ∼ 85	0		9 m, 46 s
85 ∼ 105	5		91 m, 14 s
75 ∼ 105	−5		4 m, 35 s

lists the results of the experiments for evaluating the performance of the living area identification through spatial learning. As listed in Table 2, it requires less than 20 min to detect the living area with an error below 5° in all five experimental conditions. The experiments considering those five conditions are repeated five times, and the same test time is determined in all the tests. As the human being is always in the area during the experiment rather than in the actual living environment, it can be considered that the time required for determining the human body position information is acquired faster because the information about the human body position is collected earlier. In the data collected from the general indoor room, the fastest time for early learning is approximately 12 days because there are limited people in the living space. However, as 50 data values are collected in the initial learning, the area is updated so that the area information can be updated more quickly.

5.2. Temporal Variations of Living Area

The temporal sensitivity of spatial learning should reflect the variation of the living area over time. To verify this, we conducted an experiment including the variation of the living area. In the experimental environment, as shown in Figure 15, the indoor air conditioner was rotated to evaluate the spatial learning performance when there were spatial changes over time.

After activating the spatial sensing function, the experimental target (approximately one to three people) moved freely within the designated range, and initial learning was performed for approximately 1 h. The conditions for rotating the product are as follows:

• Rotate clockwise at 0° as shown in Figure 15 and counterclockwise at 100°.
• Rotation angle in clockwise: 5°, 15°, 30°, 60°, 100°
• Rotation angle in counterclockwise: 75°, 0°

A resident who uses an air conditioner usually recognizes the changes over time t and consequently responds to the added living area immediately. In addition, if the living area is changed to the unused area frequently, it can cause an inconvenience because the air current is not directed to that area when there is no sensing in the area or if there is no person in the corresponding area for a short time. To minimize the inconvenience of the user and to manage the stable fluctuation of the product, when the 350 initial data values are secured, the living area from the initial data is directly reflected in the actual living area; thus, the actual living area is changed corresponding to the living area in the 350 units of data. The logic is designed to change the judgment after seven evaluations. For this reason, to shorten the experiment time in this experiment, when rotating clockwise, whether the current living area is modified into the living area corresponding to the area depicted in the past 15 min is examined, based on the 350 values criteria, and if the living area has remained unmodified for 120 min at 100°. In each evaluation step, the angular error is based on a value that is less than or equal to ± 5° in the case of the living zone detection experiment.

As shown in Table 2, in this experiment, initial learning takes 34 min and 28 s, and if it changes to a living area from a non-living area after that, the classification error is acceptable within 10 min. In addition, it is also verified that the classification error after 91 min changed to less than the allowable range even when changing to a non-living area from the living area at 100°. Through this experiment, it was confirmed that even after the air conditioner was installed in the home, even if the furniture location or the air conditioner location was changed, it was possible to find the living area and send airflow without any manipulation.

5.3. Air Conditioning Control Based On Spatial Learning

In this research, spatial information obtained from deep learning based spatial learning was applied to the product, and smart-care logic was used to control the left and right air flow, as described in Section 2 for user advantage. To verify these benefits, experiments were conducted to measure the reduction of time taken to achieve the target temperature and the power saving rate by cooling the actual user rather than the whole area. Experiments are conducted in an environmental chamber using a standing type air conditioner. The measured values and experimental modes are as follows in

Table 3. Experiment mode—Smart-care.

Mode	Operating Angle of Right	Left Vane (°)
Normal Smart-care	0–105
Spatial Learning	Max.	20–105
Smart-care (DNN)	Min.	35–75

.

Here, the control range of the left and right vane of the spatial learning smart care is set based on the minimum operation range of 40° and the maximum operation range of 65°. The effect of actual spatial learning is expected to be largest in the minimum range. Figure 16 shows the results of the environmental chamber test performed to confirm the characteristic of the cooling area in the living space through the learning of the deep running space.

Figure 16 is a graph of the temperature and cumulative power consumption changes that occurred during the smart-care operation. The red line indicates the change in temperature and power consumption when the vanes swing with a normal smart-care function. The green and blue lines are the graphs of temperature and cumulative power consumption during smart learning operation at minimum and maximum angles, respectively. It is confirmed that the cooling speed is faster in cases where the living area is controlled rather than in the general smart-care operation. When the living area is set at the minimum angle, it achieves the fastest cooling and the time to reach the set temperature is shortened by 19.8% when compared to the normal smart-care operation. In the cumulative power graph, the slope represents the instantaneous power consumption. Figure 16 shows that the slope decreases after reaching the set temperature, when compared to the case of rapid cooling. When the set temperature is initially reached, the slope decreases rapidly, and the power consumption difference occurs. The accumulated power consumption up to the set temperature is reduced by up to 20.5% when compared to the general smart-care while cooling the living area.

Table 4. Upper Body Detection Rate.

3 Persons/60 Sheets Each, WXGA (1280 × 640) Images
Distance	Detection Rate	Total Detection Rate
1.2 m	90% (54/60)	86.6% (156/180)
2 m	90% (54/60)	86.6% (156/180)
3 m	80% (48/60)	86.6% (156/180)

lists the time required to reach the set temperature for each mode and the power consumption.

5.4. Human Body Detection

For the performance evaluation, a total of 180 image data are used for each distance of 1.2, 2 and 3 m, and the final detection rate is 86.6% for the upper body detection. The detection speed for an upper body is 3000 ms on average (1 person detection standard).

5.5. Estimation of User-Indoor Spatial Information

Figure 17 shows that, as time passes, region estimation results are gradually accumulated to form a living space representation. Currently each region is divided into 5 levels of depth. However, when it is divided into finer scales, the wind strength to the main living area can be controlled more flexibly according to the depth.

6. Conclusions

Improving energy efficiency is becoming an important issue for all electronic appliances, especially with the strengthening of global energy regulations. In particular, the air conditioner is one of the most energy consuming products in our living environment, and a number of studies have been conducted to maintain a comfortable environment while enhancing its energy efficiency. For example, sensor technology for human detection is widely adopted for adaptive controlling of air conditioners; however, depending on the sensor coverage, detection time and the installation environment of the product, it may not be enough to efficiently control the air conditioner. Therefore, it is usually used for a limited purpose only, such as detecting the presence of a person and limit the operation of the air conditioner when there are no people in the living area.

In this study, a spatial learning based smart control algorithm is designed and implemented with low resolution visual sensors to detect living space in a living room and control air conditioner in real time. Despite the limited computing resources, it correctly detects user location and applies deep learning techniques to learn about user location information and to estimate the living space. Furthermore, the proposed algorithm also smartly controls the air flow of the air conditioner based on the result of this spatial learning algorithm. The spatial learning algorithm designed with the deep learning method shows that it can identify the living region with an error less than ±5°. From the results of spatial learning, the air conditioning system was controlled to emit a major stream of air to the living area, and this showed up to 19.8° rapid cooling under the same experimental conditions. In addition, environmental chamber tests were conducted to confirm that power consumption is reduced by up to 20.5% while achieving the set temperature. However, the proposed system takes a certain amount of time to operate properly after initial installation because it needs to learn the user environment where the product is installed.

In future research, it is necessary to expand to a variety of living spaces outside the living room and to expand the spatial detection area through human body detection in various conditions of people including multiple users, and to collect the living information through the cloud and further study the spatial learning algorithm through it. We will also consider public opinion evolution based on social studies using big data.