*5.1. Benefits to Developers*

A core design intent of SPO was the ability to be modular and extensible, unlike solutions presented in previous studies [43,46]. For this reason, SPO was developed on top of modern open-source projects such as XBOS and MPCPy. Its core communication infrastructure, based on a distributed and secure message bus [68] and gRPC [70], allows developers to seamlessly replace components such as databases, optimization engines and drivers. For instance, the native XBOS data store was replaced by another time-series database [88], due to the cybersecurity requirements of the demonstration site. The use of gRPC is an innovative feature compared to other open-source projects [57,99], and it allows developers to use languages they are more familiar with or are more advantageous for a particular application. Hence, XBOS drivers have been developed in both Go and Python as

part of this project. Further, native integration with BRICK metadata schema [66] is a distinctive feature of SPO. BRICK is an emerging "open-source effort to standardize semantic descriptions of the physical, logical and virtual assets in buildings and the relationships between them" [100]. BRICK avoids hard-coding sensor names in the software, making the application more portable between buildings. For instance, instead of representing a sensor as "supply\_temp\_VAV002\_AHU01", BRICK queries can search for the ID of the supply air temperature sensor in any VAV belonging to any AHU.

During the model construction and calibration phase, the use of MPCPy was crucial to make the software more flexible. MPCPy affords the construction of multiple systems that can be quickly replaced with others, as long as the inputs and outputs stay the same. This allows rapid switching between emulated components and real components (e.g., a virtual battery with a real battery) and vice versa. Furthermore, it provides the possibility to run the optimization engine in "shadow mode" (i.e., the optimal controller computes setpoints, but it does not send them to the actual devices) and, if necessary, to quickly swap to controlling the real systems. This setup allowed debugging the system with real-time data but avoided loss of comfort or disruption of business operation.

#### *5.2. Challenges of the Real World Deployment*

The real-world deployment of SPO was a useful step to test the robustness of the implementation and understand the challenges with real systems. The lessons learned during this process are summarized in Table 5.


**Table 5.** A summary of challenges faced during the implementation and deployment of SPO.

Selecting sensors and controllers for the project proved to be the first hurdle. While there is a proliferation of new IoT and smart home devices on the market, very few met the cybersecurity and local control requirement of the project. In particular, to access the API of most smart thermostats, an Internet connection is required. This was undesirable for a microgrid that needs to work during network outages. Eventually, a BACnet thermostat was selected for installation, although its cost was significantly higher than other alternatives and it did not provide native data encryption. There is a

clear need for modern connected devices that meet the requirements of commercial buildings and microgrids. The deployment process also revealed the effect of the fast rate of change of the online services and IoT market. In the middle of the project, the weather service used to gather weather forecasts for the site had to be replaced, because its free service was terminated after a corporate acquisition. As this paper is written, the new weather forecast service was also acquired by a different company and there is uncertainty about the future of the service and API.

Precise control of refrigerator and RTU operation through these networked controllers represented another challenge. Often, the details about the local control loop (e.g., deadbands, hysteresis and other safeguard mechanisms) were not provided by the manufacturer and they had to be reverse-engineered by the research team. While these features are useful to protect the equipment, they added uncertainty to the results of each control action determined by the supervisory controller. Additional issues related to the ability to control the system emerged when it became clear that the refrigeration system was undersized and its temperatures tightly prescribed, allowing for little flexibility in its operation. Unlike typical commercial buildings, the 24-h operation of the store also added challenges and unique problems for the testing of the system deployment. Conflicts between local temperature adjustments by the staff and the setpoints suggested by the system also emerged. In the system monitoring process, unexpected data were often observed. In one instance, the store temperature kept dropping even though the cooling systems had not been running . Analysis revealed that the door openings had a large impact on the thermal conditions of the store. This type of event was very difficult to characterize in the model. The placement of the sensors in the store also impacted the controls. As additional sensors were not installed in the store, the measurements from the temperature sensors located within the thermostats and the refrigeration controllers were used to determine the state of the system. Unfortunately, these devices were installed away from the areas where staff and customers typically orbit, and hence did not capture the user comfort.

As it is typical of deployment projects, the research team had to work with multiple departments within the same organization, each having their own set of requirements. Building occupants expressed concerns about temperature oscillations during experiments. The IT department defined strict cybersecurity requirements and access control procedures. These issues were resolved through clear and responsive communication with all parties involved at all times, providing frequent and regular updates and expectations. Providing the local staff with a real-time dashboard and alarm/notification system was a very effective way of keeping them engaged and updated. However, these needs also impacted the number of experiments allowed and their boundaries.

Further, the overall project schedule was significantly delayed because of logistical issues (i.e., unforeseen delays in equipment delivery and faulty equipment) regulatory requirements (i.e., working with utilities to sign off on the relatively novel microgrid interconnect agreemen<sup>t</sup> and receiving approval before battery and PV commissioning) and unfortunate and unforeseen natural disasters (i.e., power shutoffs due to wildfires and the COVID-19 pandemic). Although this combination of circumstances was unique, field deployments of advanced technologies should account for delays in their timeline. The delays that were experienced due to the wildfires and the pandemic only underscore the value of research that is focused on accelerating deployment of these systems. While it is unreasonable to expect that every microgrid would face similar challenges, a core goal of deployment-focused research must be to identify solutions to these challenges and improve the ability of developers to deploy advanced energy systems at scale.

#### *5.3. Limitations and Future Work*

During field tests, the SPO system demonstrated the ability to respond to both price- and and demand- based grid signals, using all the controllable loads: the battery, the two RTUs, the freezer and the refrigerator. The system has been collecting data from the local controllers and sensors for almost a year and has been controlling the equipment for seven months at the time of writing this paper. However, thus far the experiments have lasted approximately one week at a time and

the results presented in the paper covered only a single day of operations. Thus, there needs to be further prolonged testing to evaluate the performance and the robustness of the system in the long run. Additionally, the baseline load for each of the experiments has been determined using a linear regression model based on historical environmental and building load data. In addition, the focus of this paper is to demonstrate an integrated software solution in a real building rather than precisely investigating the performance of the optimal control algorithm . The next steps in the research project involve determining more accurate baselines (statistical techniques such as Random Forest Regression, Autoregressive integrated moving average (ARIMA), etc.) and improving and evaluating the performance of the optimization algorithm by comparing it against other advanced control solutions.

Currently, the measurements recorded by the temperature sensors that are embedded into the thermostats and the refrigeration controllers have been used by SPO's optimization engine. While this characteristic allows easy deployment of SPO, there has been work planned to improve the building model by collecting data from temporarily installed temperature loggers across the store. Installing additional sensors to record occupancy related information would also be greatly beneficial. For example, with a door sensor, SPO would have been able to track the door open/close events and the model would have been able to take into account the effect of outside air in the store.

While this pilot site has unique characteristics, this test successfully demonstrates the feasibility of the SPO system for similar types of buildings. Many convenience stores in operation today (around 12,000 in California alone) have similar HVAC and refrigeration systems and can be upgraded to become a building-scale microgrid. Quick serve restaurants, hotels and grocery stores are other possible candidates for deployment of these systems. Deploying SPO in such buildings would provide data points regarding the portability and the scalability of the whole system and identifying another deployment site is also a part of the future work.
