Simulation-Supported Testing of Smart Energy Product Prototypes

Abstract

Smart energy products and services (SEPS) have a key role in the development of smart grids, and testing methods such as co-simulation and scenario-based simulations can be useful tools for evaluating the potential of new SEPS concepts during their early development stages. Three innovative conceptual designs for home energy management products (HEMPs)—a specific category of SEPS—were successfully tested using a simulation environment, validating their operation using simulated production and load profiles. For comparison with reality, end user tests were carried out on two of the HEMP concepts and showed mixed results for achieving more efficient energy use, with one of the concepts reducing energy consumption by 27% and the other increasing it by 25%. The scenario-based simulations provided additional insights on the performance of these products, matching some of the general trends observed during end user tests but failing to sufficiently approximate the observed results. Overall, the presented testing methods successfully evaluated the performance of HEMPs under various use conditions and identified bottlenecks, which could be improved in future designs. It is recommended that in addition to HEMPs, these tests are repeated with different SEPS and energy systems to enhance the robustness of the methods.

1. Introduction

Within the framework for the development and implementation of smart grids, smart energy products and services (SEPS) are set to play a key role. SEPS are solutions “expected to support the active participation of end users in balancing energy demand and supply in the electricity network” [1] by creating an environment where energy use is flexible [2,3,4], efficient, reliable [5], sustainable and cost-effective [6]. Examples of SEPS include smart meters, smart appliances, electric and fuel cell vehicles [7,8], residential energy storage systems [9,10], and home energy management systems (HEMS) [11] among others.

The widespread implementation of SEPS in smart grids could enable greater interaction between end users, home appliances and energy suppliers, facilitating energy efficiency, local production and energy trading with the grid in order to improve the effectiveness of demand response strategies and reduce the required capacity for local energy storage [12,13]. This requires more active end user involvement which is currently limited by user acceptance, with users frequently finding SEPS difficult to understand and interact with [14,15,16]. Therefore, there is a need to develop more innovative SEPS that facilitate this role by achieving a better match with user expectations and demands.

Testing methods such as co-simulation and scenario-based simulations can be useful tools for quickly evaluating the technical functioning and preferred user interaction with a new SEPS design during its early development stages. Co-simulation is a method where several subsystems are simulated independently and then coupled together to analyse the entire system and the interactions between its components. This makes it possible to quickly and accurately model complex, heterogeneous systems by using the simulating tools native to each subsystem [17,18]. The use of co-simulations for modelling smart energy systems has been explored in the literature [18,19,20] but there is still little evidence of its application with SEPS prototypes. Scenario-based simulations, on the other hand, can use existing energy profile datasets to replicate real-life conditions without the need to carry out field tests, although it is important that the used data accurately reflects system behaviour observed in practice. Relevant examples of these tests applied to smart energy systems include the scenario-based simulations found in [21,22,23] as well as the user tests presented by [24,25].

In this paper, the performance of three home energy management product (HEMP) concepts are tested using a simulation testing environment as well as scenario-based simulations; results from end user tests are presented as well, serving as a validation for the modelled scenarios. The article is structured as follows: Section 2 introduces the methodology followed for each of the proposed testing methods, and the results of each test are presented in Section 3. In Section 4, a discussion on the effectiveness of these methods is presented followed by some conclusions on this study.

2. Materials and Methods

Prototypes for three conceptual HEMP designs were specifically developed for this study to serve as user interfaces for home energy management. They consisted of devices with visually appealing forms that measure energy production and consumption data from a household smart meter and display basic information to users through simple, intuitive visual feedback such as LED colouring and brightness. This feedback is updated on regular intervals to indicate how household performance changes through time and in response to users’ actions. The three developed concepts (shown in Figure 1) are described below:

1. Bodhi: An arrow-shaped “energy budget” indicator that shows users how a household’s energy use compares to a predetermined daily or weekly budget through LED colouring. The relationship between actual and planned consumption during a given interval is determined through a budget ratio (R_B, unitless) defined as:

R_B = E_cum/(j/N) * B,

(1)

where E_cum (kWh) is the cumulative energy consumption in the current period (e.g., a day or a week), j is the interval number (unitless), N is the number of intervals in a period (unitless), and B is the total energy budget for a given period (kWh). An R_B value between 0.95 and 1.05 indicates that users are “on budget” (corresponding to purple LED lighting); values greater than 1.05 correspond to an “over budget” state (orange lighting) while values below 0.95 indicate the household is “under budget” (aqua lighting).

2. CrystalLight: A smart home ornament that acts like a virtual energy storage system; each day, electricity produced by a household’s PV array makes its LEDs grow stronger (“charging” the ornament) while electricity consumption gradually dims them. The charge (C_i, kWh) and state of charge (SOC_i, unitless) for this “battery” at each measurement interval are calculated as:

C_i = C_i−1 + E_P − E_C

(2)

and:

SOC_i = C_i/C_MAX,

(3)

where E_P (kWh) and E_C (kWh) are the produced and consumed energy during a given interval and C_MAX denotes the total battery capacity (kWh). The battery’s state of charge is converted into a brightness value between 0 and 100% for the prototype lights; if at any given interval C_i becomes negative, it will be automatically set to zero to simulate an “empty” battery. Likewise, if the state of charge becomes greater than 100%, the charge will be set to C_MAX to simulate a “full” battery.

3. LightInsight: A small cylindrical dial that gives users information on the balance between a household’s energy production and consumption during the day through LED lighting. Four different feedback states are possible: net energy production (green lighting), net consumption (red), transition from green to red (yellow) and transition from red to green (rainbow). These states are determined by an energy ratio (R_E, unitless) defined as:

R_E = E_P/E_C,

(4)

where E_P (kWh) and E_C (kWh) are the produced and consumed energy during a given interval.

The autonomous operation of all prototypes was made possible through the use of Raspberry Pi microprocessors, where a Python script was created to periodically obtain energy data from a smart meter, calculate the required key indicator (R_B, SOC_i or R_E) and set the LED properties accordingly.

2.1. Simulation Environment Testing

A series of short testing sequences were developed to validate prototype operation using the simulation environment from the Smart Electricity Systems and Technology Services laboratory (SmartEST Lab) at the Austrian Institute of Technology (AIT) [26]. In these tests, energy production and consumption were independently simulated to model different system states, which were interpreted by each prototype in order to set its LED properties accordingly, as seen in Figure 2 below. This was achieved through the following process:

• Energy generation was modelled using a DC voltage/current source, which simulated a residential PV system. Energy consumption, on the other hand, was modelled using an RLC controllable load, which consumed the generated power or drew power from the local grid whenever consumption exceeded generation.
• The laboratory’s main measurement system integrated these two inputs and periodically passed them on to the HEMP prototype using the communication infrastructure, which consisted of a custom-built middleware application linking these components.
• The prototype calculated the key indicator’s new value and set the corresponding LED properties.

2.2. Scenario-Based Simulations

The operation of each prototype was further tested by using existing production and consumption datasets to model several use scenarios. Four different scenarios were created by combining summer and winter load curves with PV production data reflecting “adequate” or “inadequate” performance according to weather conditions; all sources have 1-min resolution and cover a 24-h period as seen in Figure 3 below. The following scenarios were modelled:

• Summer Load Profile, Inadequate PV Production
• Summer Load Profile, Adequate PV Production
• Winter Load Profile, Inadequate PV Production
• Winter Load Profile, Adequate PV Production

Since the prototypes were designed to periodically read energy data from household smart meters, a Python script was created that replicated this process. A series of data points, consisting of a pair of values for resp. energy consumption and energy production was modelled with this script serving as the main input for the prototype’s feedback algorithm, see Figure 4.

2.3. End User Testing

In order to compare the results from scenario simulations to a real-life situation, two of the prototypes were briefly tested with end users in several households in the Netherlands. The tests were conducted in two phases:

● Phase 1—Reference Measurements

This phase was used to create a benchmark for evaluating the effectiveness of each prototype during the second phase. Household energy production and consumption were measured on 15-min intervals by connecting a Raspberry Pi unit directly to the household’s smart meter.

● Phase 2—HEMP Prototype Testing

In this phase, users were presented with a brief description of the prototypes as well as a short demonstration of their operation, after which one of the prototypes was installed in their home. Users were then left to freely interact with the prototype for several days; during this phase there was constant monitoring of energy consumption and generation with the prototypes capturing data from smart meters at 15-min intervals.

3. Results

3.1. Simulation Environment Test Results

3.1.1. Bodhi

Figure 5 shows how the prototype’s lighting reacted to a gradual increase in cumulative energy consumption relative to an arbitrary energy budget, going from the under budget state (left) to the on budget (centre) and over budget (right) feedback states.

3.1.2. CrystalLight

Figure 6 shows different stages of the modelled charge-discharge cycle, where the prototype’s LED brightness gradually increased before reaching its maximum intensity level, then becoming dimmer until the full discharge state was attained.

3.1.3. LightInsight

The four system states for this HEMP were tested by first increasing the value of R_E from 0.9 to 1.1 (Figure 7, pictures 1–3) and later decreasing it (Figure 7, pictures 3–5) back to its initial value.

3.2. Scenario Simulation Results

3.2.1. Bodhi

This prototype was tested in two scenarios, corresponding to summer and winter loads since its operation is not dependent on PV production. In the summer scenario, a smooth transition through all three feedback states was observed, with energy consumption starting significantly under budget (R_B < 0.95) and staying on budget for a short interval before remaining consistently over budget (R_B > 1.05) for the rest of the day. This is partly due to the shape of the budget ratio curve itself, which shows an upward trend with short intervals where R_B sharply increases. These intervals match peaks in household load and are followed by gradual decreases as energy use reverts back to the baseline load. The selected energy budget (B) also had a significant impact on when these transitions took place since it determines the balance point between actual and planned consumption (R_B = 1).

The winter scenario showed a similar trend for the budget ratio throughout the day while showing more pronounced increases in R_B than during the summer scenario, as seen in Figure 8 below. Once again, the energy budget was exceeded by the end of the day, although this occurred much later; as was the case before, this greatly depended on the selected energy budget.

3.2.2. CrystalLight

During Scenario 1, this prototype spent the vast majority of the day at full discharge, only charging during a few short intervals between 7:30 and 11:00 where the maximum charge, set at 15 Wh, was quickly reached and then consumed. This should not be surprising considering that energy consumption consistently outperforms production in this scenario.

In Scenario 2, fast charging took place from 7:00 to 10:00, with the prototype fully charged for around five hours before gradually discharging for the rest of the day as seen in Figure 9a). As expected, performance was significantly better than in the previous scenario; the only times in which a full discharge occurred were the early morning hours where PV production had not yet started.

The combination of poor PV production and high energy demand in Scenario 3 resulted in the prototype being fully discharged for the entire day; this means that from the user’s perspective the product lights would be constantly off.

Finally, in a similar way to Scenario 2, in Scenario 4 the prototype went through a charge-discharge cycle during the daytime, with a second, shorter charging phase in the early afternoon (see Figure 9b below). The discharge phases were faster in this case, with the battery emptying completely by 18:00. Maximum charge was set at 4000 Wh, which explains why the charging phase abruptly stopped at around 11:00.

3.2.3. LightInsight

The prototype’s “net consumption” state took place around 93% of the time in Scenario 1, the only exception being several short periods of “net production” between 6:00 and 13:00 as seen in Figure 10a below. The two proposed transition states (corresponding to “rainbow” and “yellow” LED lighting) were extremely rare, each occurring less than 1% of the time. This is due to the abrupt changes observed for R_E, which hardly fell within the transition range (0.95 < R_E < 1.05).

As expected from the increased PV production in Scenario 2, net production periods were much more frequent, amounting to around 40% of the total intervals and lasting longer on average. The energy ratio was also significantly higher both on average (R_E = 1.7 compared to 0.3 from Scenario 1) and on its maximum value, exceeding R_E = 10 on several occasions. Transition states occurred even less frequently than in Scenario 1, both accounting for only 0.9% of the total intervals.

The performance of this prototype in Scenario 3 matched the observations made for CrystalLight since the low values of R_E failed to approach the transition range and red lights showed for the entire day. In Scenario 4, however, there were a few hours around noon where the net production state occurred with little to no interruption (see Figure 10b below). Transition states were less frequent than in any other scenario, with only two yellow intervals (0.14%) and one rainbow interval (0.07%) during the entire day. The prototype performed better than in Scenario 3 as expected but a better performance than in Scenario 1 was also achieved, showing that good PV production seems to have a more significant impact in this prototype’s feedback than changes in household consumption.

3.3. End User Testing Results

3.3.1. Bodhi

The performance of this HEMP revealed that the selected energy budget, which was based on the household’s average consumption during the previous week, greatly overestimated the actual energy use during testing. Consumption during the morning of the second day was much higher than expected but then sharply decreased, transitioning through all three feedback states and remaining on the under budget state for the rest of the day and the next two full days as well. Due to the short length of the testing period, it is hard to determine whether this decrease in consumption can be attributed to the users’ reaction to the prototype or if there was influence from other factors.

Despite this overestimation, it is still possible to see that most daily budget ratio profiles follow a similar pattern consisting of an overall increasing trend with short intervals where R_B increases sharply. As was the case with the profiles observed in the scenario simulations, these intervals match peaks in household load and are followed by smaller gradual decreases as energy use reverts back to the baseline load. A noticeable exception to this pattern was observed in the first half of the second testing day where a decreasing trend took place, as shown in Figure 11.

3.3.2. LightInsight

The performance of the LightInsight prototype during user testing showed strong similarities to the patterns observed in some of the simulated scenarios, with drastic changes to R_E taking place in brief periods of time (see Figure 12 below). Peaks are significantly more pronounced in some days than in others, possibly indicating sunny or overcast weather.

The prototype showed red LED lighting for most of the time, with scattered periods of net production appearing mostly during midday and the early afternoon (11:00–17:00). The net consumption state constituted around 86% of the total intervals; the yellow and rainbow transition states, at 1 (0.2%) and 2 (0.5%) intervals, almost never occurred during testing.

4. Discussion and Conclusions

The operation of the three HEMP prototypes was successfully tested using a simulation environment, proving the usefulness of this tool for quickly and accurately validating the operation of SEPS using simulated PV production and load profiles. The tests presented a simple, quick visualisation of how these prototypes would operate in households without the need to involve the end users themselves. This approach can be useful during the early product development phase for rapidly testing several modes of operation and determining which one is best suited for achieving the intended purpose of a given SEPS.

Although the proposed testing sequences were relatively simple, they provided a clear demonstration of how the HEMPs would operate in practice, and there is potential for improving the accuracy of these tests by designing more complex testing sequences. Future simulation testing of the presented HEMP concepts should also explore the potential for incorporating other methods such as co-simulation in order to obtain more accurate results. These tests would require the development of an agent-based model of product influence on user behaviour and the resulting impact on the residual load profile in order to upscale the physical device to a large number of simulated devices.

The scenario simulations and end user testing served as a more extensive test on HEMP performance, which helped identify some of the advantages and limitations of the current designs. For instance, tests on the Bodhi concept helped identify a recurring daily R_B profile but were limited by inaccurate predictions for the household’s energy budget, highlighting the importance of correctly estimating this type of parameter during its operation. LightInsight, on the other hand, was able to present simple, intuitive information about how energy flows in a household, but was found to be useful for only a small fraction of the day since by definition net consumption takes place whenever the sun is down. The rapid fluctuations observed in the energy profiles also meant that the proposed transition states were extremely rare, meaning they should be restructured to better respond to the observed user behaviour. Overall, the proposed feedback algorithms required only basic energy data in order to determine feedback to users. However, developing more complex algorithms that use other variables as inputs and respond to changes in use patterns could help improve the effectiveness of these concepts in the future or even add new functions, such as the scheduling algorithm for smart appliances presented in [27].

The scenario-based simulations were also intended to replicate the conditions observed during the end user testing. User test performance for the LightInsight concept was more closely approximated by Scenario 1 with a root mean square error (RMSE) between both datasets of 1.58 followed by Scenarios 3, 4 and 2 (RMSE = 1.61, 1.96 and 2.43, respectively). For Bodhi, the summer scenario (RMSE = 0.31) matched the user tests more closely than the winter scenario (RMSE = 0.35). Overall, while the simulations approximated some of the general trends observed in the user tests, they are still far from being a significant predictor of SEPS operation. Comparing HEMP performance during user tests to their reference measurements, on the other hand, revealed that the concepts did not always seem to achieve their intended purpose. Testing on LightInsight resulted in an increase in both the average load (25%) and the peak load (3%) compared to the reference phase, and a more deficient match between energy supply and demand. User tests with Bodhi, on the other hand, had a positive impact since the overall energy consumption showed a significant decrease averaging 27% less than in the reference phase. Thus, these preliminary results can show at a glance whether a given SEPS concept is working adequately, and this information can be used to modify its design during the next development phase.

It is important to consider that the simulated scenarios covered a short period of time due to limited data availability; the use of larger datasets for a longer period of time could provide a more accurate representation of the modelled operating conditions. A study by van Dam et al. [11], which sought to evaluate the effectiveness of HEMS in saving energy in households, reinforces this notion since it had a much larger sample size and a longer duration than the present work. However, the study still indicated the need for conducting more extensive research into long-term effects, and also pointed out that the initial effectiveness of HEMS feedback tends to wear off with time.

Overall, the presented testing methods were successful in evaluating the potential of HEMP concepts and identifying possible challenges or bottlenecks in their design, offering valuable insights that can result in significant early improvements and make the product development process more efficient. The presented simulation environment can be a good first approach for testing and showcasing the operation of SEPS designs in a shorter period of time than with real user tests, with the possibility of obtaining more accurate results in the future through a more extensive co-simulation approach. In addition to this, scenario-based simulations using existing energy profiles can provide an accurate approximation of real-life conditions, revealing flaws or limitations which would otherwise come to light later on and would become much more difficult to overcome. Since only a limited number and a specific type of SEPS were tested in the present work, it is recommended that these tests are repeated with a wider range of SEPS concepts as well as with different energy system configurations to enhance the robustness of these methods.