**4. Results**

This section shows, evaluates, and interprets the results of the HERMES system. It is developed under the proposed MO model integrating the Assistant.

### *4.1. HERMES System Deployment and Infrastructure*

The system has been deployed in a single-family house with four residents with an average annual pre-installation consumption of 6346 kWh and powered exclusively by the electricity grid. The deployment of the HERMES system was carried out in several phases taking into account the characteristics of the house, which has a kitchen, living room, three bedrooms, attic, and three bathrooms.

In the first phase, the passive elements of the house that could affect comfort and thermal insulation were analyzed, something basic but usually a factor that is not taken into account in most installations with HEMS [5]. Some thermal leaks were detected that could be easily solved, such as installing a weather strip (see Figure 9) on the access door to the house, improving the thermal insulation from the outside.

**Figure 9.** Weatherstripping to improve the thermal insulation of the house.

Several groups of sensors, actuators, and various smart home hubs were deployed to achieve efficient control of consumption and comfort in the home. While one group of sensors collected environmental data such as temperature, humidity, or lighting, the second group of sensors collected data on the presence or door opening, and a third group collected data on loads such as power or energy. As for actuators, there were smart switches and sockets for load control. In addition, some appliances already incorporated IoT management. Finally, the various Smart home hubs allowed communication with all sensors and actuators covering various protocols: WIFI, Z-Wave, Zigbee, BLE, and Infrared.

Following the architectural model given in Figure 1, Figure 3, and Figure 5, a comprehensive infrastructure of devices and systems was deployed for the physical implementation of the HERMES system, as shown in Table 3 and Figure 10:


**Table 3.** IoT sensors and actuators in the main loads of the home.

**Figure 10.** HERMES system infrastructure.

In addition to the sensors and actuators indicated in the table above, the system has sensors for presence, temperature, humidity, twilight, outdoor weather station, door opening in certain rooms and windows, and general consumption meters (energy and power) in the house, as well as consumption meters in certain appliances and meters in two additional areas of the house (lighting + plugs and kitchen).

The deployed infrastructure enables interoperability between devices, event synchronization, real-time (and historical) data logging, analysis and visualization, and present and long-horizon decision making by both the system and the residents, maintaining or improving comfort and cost reduction.

### *4.2. Voice Assistant and Control Panel*

The Voice Assistant provides relevant information to residents to safeguard the balance between both objectives (*f*1 and comfort) and accepts voice commands to inform or act on specific subsystems. It is the central core of communication with the residents, although they also have a control panel that offers both information (current and historical data) and the possibility of configuring most of the system parameters. The main interactions of the Wizard (Table 4), an extract of the Control Panel with options for setting some parameters (Figure 11**),** and several data access interfaces (Figures 11–13) are detailed below.

#### *4.3. Phases of Incorporation of HERMES System Functionalities and Change in Residents' Habits*

Finally, we present a series of data to conclude with the achievements in terms of consumption reduction (cost evolution graphs, load shifting, prices, consumptions, invoices) and comfort improvement (process automation, commands, automated actions).

As we will see later in the subsection "Net load shifting", residents have a wide margin of improvement for consumption reduction based on shifting controllable (and some uncontrollable) loads to hours with lower prices. The system will try to approach the state of minimum consumption while maintaining comfort. Residents are provided with more information to make decisions they might not have considered before, allowing them to achieve an optimal balance between comfort and electricity bills by adjusting the parameters to their preferences at any time. The information provided by the system through the information panels, or the Voice Assistant keeps users constantly informed of the influence of their consumption habits on their electricity bills.

**Command Type (Residents Request Information/System Informs about Triggering Events) Description** "turn on/off/regulate device" Residents/System Residents control more than 80 functions (turn on, turn off, raise the temperature by one degree) of the different devices connected in the home. In some cases, the system detects that a device has been switched on so that under certain conditions, it acts automatically to reduce consumption while maintaining comfort (e.g., it raises the cooling temperature by one degree after a few minutes of operation). "price", "power", "consumption", "daily consumption", "cheapest washing machine/hour"... Residents Residents can ask at any time for data related to consumption and expenditure: Price or active power being consumed at that moment to know the impact of connected appliances, the next cheapest hours, the accumulated consumption per hour, daily or monthly. "room temperature", "outside temperature", "thermos temperature", "probability of rain".... Residents can know the data from the sensors connected in the house through the voice assistant or the probability of rain to make decisions based on these conditions and the electricity tariff to reduce consumption and maintain comfort. "Departure or arrival home" (GPS + ping Wifi + door sensor). System The system detects if a Resident arrives or leaves the house by issuing a welcome message or checking if there are devices or unwanted presences. "Power warnings" System The System monitors the active power level, informing Residents if the contracted power limit is reached or exceeds 105%, which would incur penalties. "Price and consumption/expense notices" System The System reports at each start of a time slot with a different energy price, except during night hours (peak, flat or off-peak). In the event of higher or lower than expected consumption, the reports and responses to automatic warnings and queries made by the Residents to the Assistant are modified. "Notices on ways to save" Residents/System A compendium of tips with saving techniques. The advice offered is random unless an inappropriate use of an appliance is detected (e.g., forced turning on of the electric boiler or continued use of the washing machine during peak rate hours). The system has a calendar, so some responses and warnings change depending on whether it is a national holiday or a weekend, or if adverse weather conditions are expected, or a very high consumption prediction estimated by ML. "Text-to-speech and social networks" Residents/System Residents can send any command to the Assistant through their mobile application by voice commands or by text through social networks that the Assistant receives and executes. The System uses social networks to send text and text-to-speech messages to Residents withthehelpoftheAssistant.

**Table 4.** Main voice interactions with the Assistant.

**Figure 11.** Control Panel excerpt (OpenHAB).

**Figure 12.** Data dashboard extract (Grafana + InfluxDB + MariaDB).

**Figure 13.** Example of text messages and images sent by the system to residents (Telegram).

The following figures (Figures 14–18) show how the daily distribution of loads has changed in line with prices and the impact these changes have had on bills. Both the system and the residents have been adapting to each other to achieve the above-mentioned optimal balance. The data have been divided into four periods (see Table 5): (0) P0 or previous. (1) P1 or first period where the system was still being implemented, and the optimal tariff was determined according to the residents' habits and HERMES' potential. In this period, the system did not ye<sup>t</sup> allow load shifting, but it did offer information on their consumption. It was determined that it was necessary to move from the 2.0A tariff without time discrimination to the 2.0DHA tariff, distinguishing two price bands. (2) P2 or the second period starts with the new tariff, consumption managemen<sup>t</sup> and allows the displacement of some loads. (3) P3 or third period where the system is implemented with the total operational capacity to displace all dispatchable loads and is ready to readjust the cooling/heating temperature to optimize consumption and comfort managed by the Wizard.

**Figure 14.** P0 and P1. (**a**) Average consumption (Wh) and (**b**) average prices (€ cents per kWh) during P0 and P1 (tariff 2.0A).

**Figure 15.** P1. (**a**) Average consumption (Wh) and (**b**) average prices (€ cents per kWh) during the first period (tariff 2.0A).

During P1 (first period), the HERMES system infrastructure was developed and started to work effectively from P2 (second period), with full development in P3 (third period). During P1, the residents already have information on their consumption, but the system cannot shift loads. The maximum consumption coincides with the most expensive hours. The pattern of P2 and P3 is very different from that of P1 (see Figures 14–18), mainly due to the shifting of loads to the cheapest price hours, optimizing the monthly electricity bills as shown below (see Figure 24). Consumption has shifted from being centered from 17 h to 20 h, coinciding with the most expensive hours, to being divided into two and three bands of specially reduced prices, centered from 02 h to 04 h, from 10 h to 12 h, and 23 h, coinciding with the average of the lowest prices. Above all, this adjustment

stands out in the third period, where the load shifting is optimized to reduce the bill while maintaining comfort, being very significant to see how the consumption needs are reduced in the most expensive hours (from 19 h to 21 h) and concentrated in the cheapest ones while maintaining a certain balance due to the maintenance of the residents' comfort.

**Figure 16.** P2. (**a**) Average consumption (Wh) and (**b**) average prices (€ cents per kWh) during the second period (tariff 2.0DHA).

**Figure 17.** P3. (**a**) Average consumption (Wh) and (**b**) average prices (€ cents per kWh) during the third period from 29 March 2020 to 24 October 2020 (tariff 2.0DHA).

**Figure 18.** P3. (**a**) Average consumption (Wh) and (**b**) average prices (€ cents per kWh) during the third period from 25 October 2020 to 06 February 2021 (tariff 2.0DHA).


**Table 5.** Phases of incorporation of HERMES system functionalities.

### *4.4. Net Load Displacement*

It would be necessary to compare the real load distribution with respect to a scenario with no load shifting and no change inhabits to quantify the savings provided by the Hermes system. From the recorded data, two scenarios can be distinguished, one formed by periods P0 and P1 in which there were no load shifts or changes in habits, and another scenario formed by periods P2 and P3 in which HERMES has carried out load shifts, and there is some adaptation of the residents' habits to the time slots with lower prices.

From the first scenario (no-load shifting and no change in habits), an "average load distribution" has been obtained for each hour of the day so that the load shifting for any given day can be calculated by obtaining the difference of loads to the average distribution. A distinction is made between shifts that produce savings (above average loads at economic hours or below average at expensive hours) and those that do not produce savings (below average loads at economic hours or above average at expensive hours). The difference between the displacements that produce savings minus those that do not produce savings gives us the measure of the net displacement of loads, this being positive when savings are produced and negative when cost overruns are produced, and the greater the displacement, the greater the savings, balanced by the price per kWh and total consumption, so although it offers a measure of displacement, it does not offer a direct measure of the savings that will be calculated as will be explained later. Figure 19 shows the "average load distribution" for the scenario without load shifting, and the load distribution for the day 23 October 2020 has been added as an example to obtain the net displacement for that day:

**Figure 19.** Average load distribution for each hour of the day and the scenario without load shifting or habit adaptation (P0P1 series). The load distribution for day 23 October 2020 has been added as an example to obtain the net load shifting.

The calculation of the "net load shifting" (shifts that produce savings: Add; shifts that produce cost overruns: Subtract) for that day, following the procedure indicated in the previous paragraph, the net balance is positive and has a value of 10.69 kWh:


Suppose we extend this calculation to all the days of the different billing periods (periods indicated in the first column of Table 6). In that case, we obtain the following graph with the net load shifts per billing month, obtaining an average daily net shift of 5.61 kWh for the billing range 23–37, which is equivalent to 35.8% of the average daily consumption established at 15.68 kWh (481.88 kWh for each billing month). In Figure 20, two zones can be distinguished, one with negative shifts where there were no savings and covers from invoice 15 to 21, and the other from 23 to 37 where all net shifts are positive, which indicates the correct operation of the HERMES system. Even invoice 22 already has a positive shift, although it was not enough to obtain significant savings; that month was the one in which HERMES started operating. It is also shown how during the summer months of July and August (invoices 31 and 32), the system is less efficient since the most intense use of refrigeration coincides with the most expensive hours and represents a significant part of the total consumption.

**Figure 20.** Net load shifting (kWh) per billing month. A positive net balance is obtained from billing 22 due to the performance of the HERMES system.

### *4.5. Calculation of Balanced Savings Obtained by HERMES*

Once the net load shifting has been obtained, the savings calculation will partly follow the data obtained previously, but taking into account the total consumption of each day and the prices for each hour of that day. We started from the scenario with no load shifting or change of habits, in which electricity tariffs did not influence residents' habits since the behavior pattern was based on comfort. Based on this pattern, the actual daily consumption, and the two most favorable tariffs (2.0A and 2.0DHA), an expense model is obtained for P0 and P1, as shown in Table 6. The first model, P0P1 2.0A, during billing 15 to 22 only has a mean deviation of ±0.32 € to the actual monthly billed behavior, validating its use as an estimate for subsequent billings. If we modify the model for the 2.0DHA tariff, we obtain the third column of Table 6 (P0 and P1 2.0DHA) that offers lower costs simulating

a scenario in which residents prioritize comfort but would have contracted the 2.0DHA tariff. Next, we will compare both models for actual consumption to determine the savings generated by the HERMES system after its implementation.

**Table 6.** Cost of energy consumed (€) by billing months for the scenario without load shifting or behavioral adaptation (P0P1 series) and actual billed cost. The data are divided into modeled and real data.


Actual data provided by the electric company.

1

From billing period 23 to 37, the average monthly savings in energy billed would be from 7.08 € to 11.58 €, i.e., a reduction of between 16.24% to 24.08% in energy billed compared to models without load shifting (see Table 6). If we consider taxes (excise tax of 5.11269632% for VAT of 21%: \*1.0511269632\*1.21), the average saving in each invoice

would be from 9.00 € to 14.73 € (from 0.3 to 0.5 € per day) since the implementation of the HERMES system.

If we represent these data graphically, we obtain Figure 21:

**Figure 21.** Cost of energy consumed (€) by billing months for the scenario without load shifting or habit adaptation (P0P1 series) and the actual cost billed since implementing the HERMES system.

> Finally, in Figure 22, we compare the cost of energy consumed daily for the two regulated price tariffs, tariff 2.0A and 2.0DHA, from the first period to the third period.

**Figure 22.** Comparison of the cost of energy consumed daily between the 2.0A (blue) and 2.0DHA (orange) tariffs from 17 February 2019 to 31 August 2020.

> We can see how graphically the cost in both tariffs is very similar during the first period. From the second period onwards, the 2.0DHA tariff was contracted, which implied

a change in certain habits of the residents to adapt to the new tariff. In addition, from this second period onwards, the system already managed consumption and load shifting, which made it possible to optimize the time slots with lower prices, achieving a very significant reduction in the daily cost compared to the 2.0A tariff.

### *4.6. Billing Expenses in Absolute Values without Balancing*

Independently of the studies and models discussed above, we can conclude the savings analysis by detailing the bills issued by the electricity company, although in this case, the results are not balanced against price variations (tariff 2.0DHA: 2018: 0.1025 €/kWh; 2019: 0.0898 €/kWh; 2020: 0.0739 €/kWh), different annual temperature cycles (average Tmean: 2018: 18.2 ◦C; 2019: 18.8 ◦C; 2020: 19.2 ◦C) or different annual consumptions (total per year: 2018: 6346 kWh; 2019: 5211 kWh; 2020: 5644 kWh). However, it is of interest to show them given that the variations in conditions between 2019 and 2020 have not been very significant and ye<sup>t</sup> show a remarkable reduction in bills even though the reduction in consumption has not been so significant (see Tables 7 and 8, Figures 23 and 24), mainly due to comfort requirements (higher consumption). Despite these demands, all months from the first period (31 March 2019 to 26 October 2019) present lower bills than the previous period (from 1 January 2018 to 31 March 2019), with a reduction of 18.3% where residents were unaware of their consumption details; as the first period progresses, the reduction in the bill is increasingly significant. This reduction is very striking with the entry of the second period (from 4 November 2019 to 28 March 2020). During this phase, the sum of the bills amounts to 357.1 € compared to 639.8 € during the same period a year earlier; the saving is 282.7 €, reducing 44.2% in the electricity bill. Finally, the bills from the third period (from 29 March 2020 to 31 August 2020) add up to 327.5 € compared to 427.7 € in the first period (reduction of 23.4%) or 515.3 € in the previous period (reduction of 36.5%) during those same months (from April to July), which shows the efficiency of the system able to continue optimizing periods when the system was partially implemented and already showing good performances as it was the first period. Table 7 (energy consumed) and Table 8 (monthly billing) also show the annual variations, including all periods.


**Table 7.** Monthly energy billed: previous period P0; first period P1 (yellow); second period P2 (green); third period P3 (blue).


**Table 8.** Monthly bill: previous period P0; first period P1 (yellow); second period P2 (green); third period P3 (blue).

1 HERMES System upgrades from 20 December 2020 to 11 January 2021 for system maintenance (change from Raspberry Pi3B+ to 4B, upgrade to Raspbian Buster, Java 11, OpenHAB 3, fixed and removed security bugs, update of certain parts of the programming due to version changes and new syntax...).

The following figure shows a comparison of the data in Table 7:

The following figure shows a comparison of the data in Table 8:

Since the implementation of the system (a process developed during the first period), there has been practically no reduction in energy consumption in the home (see Figure 23), so comfort has not been sacrificed. However, the electricity bill has been reduced (see Figure 24); that is, the comfort of the residents has been maintained (and even improved) (thanks to the Wizard), and the loads have been shifted to reduce the monthly bill significantly.

The Assistant has significantly improved the residents' sense of comfort by allowing them to voice-control most of the charges. Thanks to this positive impact, the additional (and primary) function of the Assistant of being able to transfer information to the residents (and to the system) to reduce the amount of the bills has been easily assimilated by the

users, so the impact has been very positive and relevant, favoring the feeling of comfort and the reduction of the electric bill of up to 42%.

**Figure 24.** (**a**) For each month, a comparison of the amount of the electricity bill (€) is sorted by year. (**b**) For each year, a comparison of the amount of the electricity bill (€) is sorted by month.

### *4.7. Consumption Estimation (Machine Learning)*

Finally, we show a comparison between what was consumed and the consumption estimate in Figure 25, which allows residents to detect habits that may increase spending when actual consumption consistently exceeds the estimate or beneficial habits when actual consumption is lower than the prediction.

For ML development, several regression algorithms were used to train the model. Given the characteristics of the data and the desired outcome, the algorithms offering the most accurate predictions were boosted decision tree regression (BDTR) and decision forest regression (FDR), as opposed to linear regression or neural network regression [93–95]. In our case, after multiple pieces of training with different data structures, the BDTR algorithm has provided excellent accuracy (coefficient of determination: 0.9842; relative absolute error: 0.1085; mean absolute error: 452.399) at the cost of moderate training times. The BDTR algorithm is very sensitive to overfitting, so care must be taken in setting up the algorithm.

The consumption forecast obtained by ML is very accurate because it handles a large number of variables, so if the same conditions are repeated, the consumption should be similar. Although there may be discrepancies, the long-term trend should show a high correlation between the forecast and the actual consumption, which made the BDTR algorithm an optimal candidate because it is based on the creation of a set of regression trees through boosting, which means that each tree depends on previous trees. The algorithm learns by adjusting the residual value of the trees preceding it, so boosting tends to improve accuracy by creating series of trees incrementally and selects the optimal tree by an arbitrary differentiable loss function.

The study data in this paper cover periods extending before, during, and after the confinement period due to COVID'19. Residents remained during the first confinement period (15 March 2020 to 20 June 2020) in the home and through mid-August 2020, with consumption increasing significantly during July due to high-temperature weather. In general, it was expected that consumers would be much higher than normal during the confinement period because the residents remain in the home all the time, which should translate into higher consumption. Figure 23 shows that the consumption from March to July 2020 is higher than the previous two years; however, the bills during that period (see Figure 24) were lower than the previous years (except July 2020). This highlights two relevant aspects, on the one hand, consumption should have increased significantly, but the system as a whole has been able to control these unfavorable conditions, and on the other hand, bills should have been much higher than in the same period of previous years, but again, the system has been able to manage the loads by reducing the energy impact to bills with lower amounts than in the previous two years. The system's efficiency is very relevant, as, without it, we could have expected these bills to have increased very significantly.

**Figure 25.** Comparison between estimated consumption and actual consumption in Wh. (**a**) ML Table (17 February 2019 to 5 April 2020) and test period (6 April 2020 to 9 September 2020). (**b**) Test period (6 April 2020 to 9 September 2020).
