Performance Assessment and Comparative Analysis of Photovoltaic-Battery System Scheduling in an Existing Zero-Energy House Based on Reinforcement Learning Control

Abstract

The development of distributed renewable energy resources and smart energy management are efficient approaches to decarbonizing building energy systems. Reinforcement learning (RL) is a data-driven control algorithm that trains a large amount of data to learn control policy. However, this learning process generally presents low learning efficiency using real-world stochastic data. To address this challenge, this study proposes a model-based RL approach to optimize the operation of existing zero-energy houses considering PV generation consumption and energy costs. The model-based approach takes advantage of the inner understanding of the system dynamics; this knowledge improves the learning efficiency. A reward function is designed considering the physical constraints of battery storage, photovoltaic (PV) production feed-in profit, and energy cost. Measured data of a zero-energy house are used to train and test the proposed RL agent control, including Q-learning, deep Q network (DQN), and deep deterministic policy gradient (DDPG) agents. The results show that the proposed RL agents can achieve fast convergence during the training process. In comparison with the rule-based strategy, test cases verify the cost-effectiveness performances of proposed RL approaches in scheduling operations of the hybrid energy system under different scenarios. The comparative analysis of test periods shows that the DQN agent presents better energy cost-saving performances than Q-learning while the Q-learning agent presents more flexible action control of the battery with the fluctuation of real-time electricity prices. The DDPG algorithm can achieve the highest PV self-consumption ratio, 49.4%, and the self-sufficiency ratio reaches 36.7%. The DDPG algorithm outperforms rule-based operation by 7.2% for energy cost during test periods.

1. Introduction

Buildings account for 30–40% of total energy usage in many developed countries [1]. In the US, buildings consume 75% of the total national electricity, and building energy consumption contributed to 22% of national carbon emissions in China in 2021 [2]. In the context of buildings’ high energy consumption, it is essential to promote the development of distributed renewable energy sources to decarbonize building energy systems [3,4]. Conventional buildings only rely on importing electricity from the public grid, while zero-energy houses (ZEHs) can use on-site energy sources as well as grid imports to meet the energy demand [5]. The basic ways to achieve ZEHs can be summarized as minimizing demand from buildings and maximizing consumption of variable renewable energy resources [6]. However, operational performances of existing distributed ZEHs usually perform poorly, including inadequate energy flexibility, low renewable energy consumption ratio, and high operating costs. In addition, the issues of renewable energy sources consumption and real-time balancing between supply and demand have arisen. Renewable energy sources are intermittent and uncontrollable, and building user loads are time-varying. This mismatch between supply and demand limits the local consumption capacity and penetration of renewable energy [7,8]. These issues also pose challenges to the reliability of grid operation, especially while selling excess generated power to the grid [9]. It aggravates the fluctuation in the net load of the grid (i.e., the grid load minus renewable energy generation) [10]. The energy storage system can mitigate the mismatch between on-site energy supply and user demand [7]. In the renewable energy system of a ZEH, the proximity of renewable energy sources, load, and storage units makes it easy to coordinate the production, storage, and demand of energy at different times through real-time control, which promotes the local consumption of renewable energy. Results of previous works indicate that there is potential to increase cost-effectiveness [11] and local consumption performance [12] through real-time smart control. Real-time control for building energy systems has emerged as a promising alternative to improve operational cost-effectiveness and local renewable energy consumption performances [1,13].

1.1. Literature Review

1.1.1. Building Energy System Management

The rising penetration of distributed renewable energy resources has transformed a large number of buildings into active energy prosumers [6]. Activating the flexibility potential of demand-side resources is crucial to improve on-site PV consumption and cost-saving. Demand-side management (DSM) can be defined as the end-user behaviors that efficiently manage energy usage [14]. DSM can reduce energy consumption or cost through load shifting [6], load modulation [13,15], and other demand response (DR) strategies. Chia-Shing Tai demonstrated the potential for grid load shifting by implementing DSM strategies [16]. Additionally, researchers have focused on achieving flexible DR, particularly through controlling heating, ventilation, and air conditioning (HVAC) loads [17,18]. Max Bird et al. proposed HVAC control based on a model predictive control (MPC) algorithm. Results show the proposed approach can achieve an improvement of 1.7% cost saving while ensuring thermal comfort compared to the baseline [19]. Rui Tang et al. achieved a 19–22% energy reduction by modulating the demand power of a building’s chillers to improve indoor thermal comfort [20].

In recent years, the academic community has placed considerable emphasis on optimizing control actions in building energy systems to participate in DSM. Prateek Munankarmi et al. controlled building loads, PV, and battery systems based on a home energy management system to minimize utility bills while maintaining occupant comfort. The bill of a ZEH pays approximately 22% less in the simulated results [21]. Lissy Langer et al. utilized real-time control strategies to optimize the operation of residential building energy systems with PV and heat storage. The energy systems achieved self-sufficiency at 79% and 80%, respectively [22,23]. Moreover, a long-term study conducted by Kun Zhang et al. demonstrated that effectively controlling the building energy system could lead to annual electricity cost savings of 12% and a 34% reduction in peak load demand while maintaining indoor thermal comfort [12].

1.1.2. Challenges of Building Energy System Control

Rule-based strategies have become popularly used for real-time control of building energy systems. Jafari et al. proposed a hybrid approach consisting of rule-based real-time controllers and dynamic planning-based optimization techniques for managing building energy systems. Their research found that energy costs for building users dropped by an impressive 85% compared to a grid-based energy supply [24]. Though traditional rule-based control may be effective for some operational scenarios, it relies heavily on prior experience for parameter setting and requires deterministic control strategies specific to each building. Additionally, the rule-based approach has difficulty handling the uncertainties in building energy systems.

MPC has been involved in the field of building energy system control [17,20]. MPC is a constrained optimal control strategy that uses a prediction model of the building energy system to predict its future behavior. The central feature of MPC is that it can forecast the future evolution of the system over a rolling timescale [25]. MPC can ensure optimal control actions based on the optimization goals using these forecasts. The prediction model can be physics-based (white box), hybrid (gray box), or data-driven (black box) [1]. White boxes use physics information to describe the building energy systems. They emphasize the requirement for the accuracy of the physical model. The accurate physical model highlights the requirement for expert knowledge of users [26]. Additionally, the building energy system environment is often complex and variable. MPC models cannot be dynamically adjusted to adapt to the changes in the environment, leading to the increasing inaccuracy of the physical model [27]. In the existing research, many prediction models of MPC are white boxes. A number of large commercial office buildings successfully operate a cloud-based real-time white box MPC in the practical aspects [26]. Black boxes are based on data-driven models combined with machine learning [28] or statistics. However, few models discuss the applications of MPC based on black boxes in building energy systems [29]. Because of these issues, MPC algorithms for building-distributed energy systems are limited to broad implementation.

1.1.3. Reinforcement Learning

With the development of artificial intelligence techniques, the data-driven RL approach has emerged as an attractive alternative to MPC for optimizing energy system operations [23,30]. Pre-determined strategy and accurate physical models are not required in RL algorithms. RL learns the optimal control policies from interactions with the environment and selects the actions based on a given reward mechanism [27]. The control policies can be adaptively adjusted based on the feedback from the environment and show an advantage in adapting to the stochastic environment changes [31].

Table 1. Relevant review works of energy system operation based on RL.

Ref. &Year	Algorithm	Object	Model	Control Action	Reward	Purpose
Sara Abedi et al., 2022 [32]	Q-learning	Residential energy management	Model-free	Charge and discharge power of battery	Energy cost drawn from the grid	Storage efficiency, energy cost saving
Mohammed H. Alabdullah et al., 2022 [33]	DDQN	Microgrid energy management	Model-free	Power of the generation system, charge and discharge power of the storage system	Operational costs and penalty for violating power flow constraints	Calculation efficiency and operational cost saving
Rendong Shen et al., 2022 [34]	DDQN	Building energy management	Model-free	Airflow ratio, battery state of charge	Energy cost, unconsumed amount of renewable energy, thermal comfort	Equipment control considering comfort
Zhiqiang Wan et al., 2018 [35]	DQN	Residential energy management	Model-free	Electricity from grid	Energy cost and penalty for violating the allowed minimum SOC	Power cost saving
Sepehr Sanaye et al., 2021 [23]	DQN	Residential area energy management	Model-based	Load of combined heat and power generation, heat from gas water heater	Total operational cost at current time interval	Energy cost saving, self-sufficiency ratio
Alexander Dreher et al., 2022 [36]	PPO	Wind power and Hydrogen storage system	Model-based	Electrolyzer power	Electrolyzer applicable operational set point, power demand of the hydrogen compressor, amount of hydrogen, and natural gas utilized	Opportunity cost
Samir Touzani et al., 2021 [30]	DDPG	HVAC system and battery system	Model-free	Air supply temperature and flow, charging and discharge power of battery	Energy purchased from the grid/Energy cost, the penalty for the violation of the temperature comfort zone, and battery physical limits	Energy cost saving considering thermal comfort
Yuan Gao et al., 2022 [37]	DDPG& TD3	Office building energy management	Model-free	Continuous control of biomass generator, PV and battery	Amount of electricity purchased, and penalty for violating the given minimum and maximum SOC	Off-grid operation and battery safety

summarizes different applications of RL algorithms in managing energy system operations. RL algorithms can be divided into model-based and model-free RL algorithms. Model-free RL agents learn the optimal policies through continuous experimentation without using prior knowledge regarding system dynamics. Enormous amounts of data and time are necessary for high-quality convergence effectiveness [27]. Model-free algorithms perform well in an environment where the dynamics change stably. However, real-world data with stochastic patterns generally lead to an unstable training process. Model-based algorithms take actions obtained based on a simple environment model, leading to low requirements for data and better generalization of convergence [31].

Most researchers have been attempting to optimize objectives (e.g., user cost reduction, renewable energy consumption, user comfort, and load flexibility) by implementing a single approach, such as the rule-based method, Q-learning algorithms, DQN, and other regulation strategies. Thus, it is necessary to analyze the strengths and weaknesses of each RL algorithm in detail. This emphasizes the importance of further discussion and investigation to compare the effectiveness of different RL algorithms in energy system operation.

1.2. Research Contributions

The main contributions of this work are summarized as follows:

• We formulate the operation problem of a hybrid energy system in ZEHs as Markov decision processes (MDP). Additionally, we adopt model-based RL algorithms to optimize the operation problem and design an appropriate reward function to guide the energy system operation. The proposed reward function considers the physical constraints of the battery and the energy cost.
• Measured data of a ZEH are applied to train and test the RL agents. The proposed approaches achieve outstanding operation performances utilizing a relatively small amount of data.
• The scheduling performances of Q-learning, DQN, and DDPG agents are compared. The study demonstrates the cost-effectiveness of the proposed RL control approach in different seasons.

2. Methodology

2.1. Reinforcement Learning

Reinforcement learning is highly regarded as an adaptable feedback controller that can respond to environmental changes. RL can achieve goal optimality in control problems while considering any uncertainty. Control problems are generally described as MDP [38]. The quaternion (s, a, r, π) is used to show the agent, and the agent can be defined through the following Equation (1):

$$Q\left(s_t,a_t\right)\leftarrow Q\left(s_t,a_t\right)+\alpha \left[r_{t+1}+\gamma \underset{a}{{\displaystyle \max }}Q\left(s_{t+1},a\right)-Q\left(s_t,a_t\right)\right],$$

(1)

where α represents the learning rate, which means the level of trustworthiness of the update section. γ represents the discount factor of the future Q value in the present. s represents the environment state, and a is the action that an agent can perform. r represents the defined reward equation according to demand. π represents the strategy of the agent, which determines the action to be taken at a given state and is the core of RL agents. The setting of parameters, such as α and γ, is an important part of a model. It influences the operation and convergence performance.

2.2. Q-Learning

The Q-learning algorithm defines the state-action value function based on target questions, state space, and control actions. This function is commonly referred to as the Q function. The agent employs the reward value to update the Q table by taking corresponding actions in different states. After several episodes, the agent can learn how to act to gain the highest accumulated reward and gain the optimum strategy collection to solve the target question. According to the Q table, the decision key involves selecting the action with the highest Q value. A significant feature of the Q-learning algorithm is its ease and computational efficiency. It usually uses discrete actions to solve problems concerning optimization and regulation with small action spaces and state spaces.

2.3. Deep Q Network

The Q-learning algorithm is suitable for solving low-dimensional state space questions. When the Q-learning algorithm is used to address practical problems, the issue of “dimension disaster” occurs. The deep learning network has significant function-fitting capabilities. The Deep Q-Network (DQN) algorithm combines the strengths of Q-learning and deep learning algorithms by replacing the Q tables with deep learning networks. These networks take the state information of the temporal sequence as input and output the Q value of each action corresponding to the state. Actions are chosen based on the ε-greedy strategy [39,40].

The DQN algorithm introduces several innovations. (1) The use of evaluation and target networks. The evaluation network learns directly from the environment, and the target network maintains the stability of the target Q value over a certain period. It benefits mitigating the volatility of the model. (2) Experience replay, where a certain number of observations (s, a, r, s’) are stored. Then, a batch size of samples is randomly selected from the replay for training. The evaluated Q value, target Q value, and reward are used to calculate losses for adjusting the evaluation network weights by feedback. The replay allows the network to obtain the weights to suit more scenarios. The loss, which is the squared error between the target Q value and the evaluated Q value, is calculated as follows in combination with Equation (1).

$$Q_{target}=r_{t+1}+\gamma \underset{a}{{\displaystyle \max }}Q\left(s_{t+1},a\right),$$

(2)

$$Q_{eval}=Q\left(s_t,a_t|\theta \right),$$

(3)

$$L\mathrm{oss}=E\left[ {\left(Q_{target}-Q_{eval}\right)}^2\right],$$

(4)

where $Q_{\mathit{target}}$ represents the target Q value and $Q_{\mathit{eval}}$ represents the evaluated Q value. θ represents the weights of the evaluated network. Loss represents the loss.

2.4. Deep Deterministic Policy Gradient

The deep deterministic policy gradient (DDPG) algorithm integrates the advantages of actor–critic algorithms in RL based on the DQN. This method allows the agent to obtain an optimal policy based on continuous action control [41]. The DDPG algorithm consists of four convolutional neural networks: actor network, critic network, target actor network, and target critic network. The actor network provides a deterministic action in the current state s based on the strategy, and the critic network displays the Q value of the action to be performed in the current state based on the value. The target actor network and the target critic network output the action a′ and Q value Q′ corresponding to the next state s′, respectively. In contrast to the DQN evaluation network, which outputs the Q values of all discrete actions in the current state, the DDPG actor–critic network shows the Q-value of a deterministic action in the current state. Equations (5)–(8) provide the calculation of the loss parameter for updating the weight of the actor network and the loss parameter for updating the weight of the critic network.

$$a_i=\mu \left(s_i|{\theta }^{\mu }\right),$$

(5)

$$loss\_actor=\frac{1}{batch \mathit{size}}{\displaystyle \sum _{i}Q\left(s_i,a_i\right)},$$

(6)

$$y_i=r_i+\gamma Q^{\prime }\left(s_{i+1},a_{i+1}\right),$$

(7)

$$loss\_critic=\frac{1}{batch \mathit{size}}{{\displaystyle \sum _i\left(y_i-Q(s_i,a_i)\right)}}^2,$$

(8)

where batch size denotes the number of samples from the replay, 1 ≤ i ≤ batch size. $a_i$, $s_i$, and $r_i$ are the action, state, and reward of a sample, respectively. ${\Theta }^{\mu }$ represents weights of actor network μ. Equation (5) represents the actor network μ and calculates the output action under the state $s_i$. Q is the Q value corresponding to $a_i$ and $s_i$, and $y_i$ is the Q value of the action taken in the current state. Loss_actor is the loss parameter of the actor network, and loss_critic is the loss parameter of the critic network.

3. Experiment Design

3.1. Reinforcement Learning Agent Design

This paper focuses on the energy system of ZEHs, comprised mainly of PV generation, a battery storage system, and user demand connected to the grid [34,35], as shown in Figure 1. Real-time electricity prices play an essential role in encouraging DR, which helps improve electricity consumption behavior in buildings. Consequently, the energy system is regulated based on real-time electricity prices to encourage effective interaction between buildings and the grid, ultimately aiming to save energy costs [42]. The control problem is described using the MDP. Figure 2 depicts the regulation process for a building’s renewable energy system based on RL. The following sections introduce the details of our proposed data-driven, model-based RL algorithms.

3.1.1. System State

The energy system model uses PV generation (kW), power demand (kW), state of charge of the battery, and real-time electricity prices as the RL observations. Thus, the state space is defined as follows:

$$s=\left[P_{pv}\left(t\right),P_{load}\left(t\right),SOC\left(t\right),R_{grid}\left(t\right)\right],$$

(9)

where t means at current time t, and $P_{\mathit{load}}\left(t\right)$ represents the user load at time t. Similarly, $P_{\mathit{pv}}\left(t\right)$ represents the PV generation power, and $R_{\mathit{grid}}\left(t\right)$ denotes the real-time electricity prices (JPY/kWh). SOC(t) is the state of charge (SOC), and the maximum and minimum values of SOC are stable, ${\mathit{SOC}}_{\mathit{min}}$ = 20% and ${\mathit{SOC}}_{\mathit{max}}$ = 95%. As it is known, the Q-learning model cannot solve high-dimensional state problems. Therefore, the state components are divided into 10 discrete grids based on their maximum and minimum values at equal intervals. The DQN and DDPG model can handle the continuous state problems by neural networks instead of Q tables. Thus, the state components do not need to be discretized.

3.1.2. Control Action

In this work, the control action is the power charging and discharging activities. Action space is described as:

$$a=\left[P_{ba}\left(t\right)|g\right],$$

(10)

$$-P_{\mathit{maxdch}}\le P_{ba}\left(t\right)\le P_{\mathit{maxch}},$$

(11)

where g represents the discretized granularity, g = 0.25, $P_{\mathit{ba}}\left(t\right)$ represents the discrete battery charge/discharge power at time t, $P_{\mathit{maxch}}$ and $P_{\mathit{maxdch}}$ are the maximum charge and discharge power (kW), $P_{\mathit{maxch}}=0{.3 \times E}_{\mathit{cap}}$, and $P_{\mathit{maxdch}}=0{.25 \times E}_{\mathit{cap}}$. The battery’s charge/discharge power limit satisfies Equation (11). To avoid a local optimum solution, the algorithm uses the more common ε greedy strategy to ensure that each action is potentially chosen. The discretized granularity affects the speed of agent training and has little effect on the quality of learning. DQN and QL agents can both handle discrete action control problems. In this study, the discrete granularity settings are consistent. The control action of the DDPG agent is continuous, so discrete is unnecessary.

3.1.3. Reward Function

The RL agent can obtain the reward after performing the actions. The RL agent’s learning goal is to maximize the accumulative reward. The reward can be expressed in certain values. The reward function guides the agent in optimization policy. Additionally, the reward function decides the value of a reward that the agent can obtain through taking each action. The value of the reward influences the learning direction of an agent.

The optimization of a ZEH energy system aims to improve on-site PV consumption and minimize energy costs. Furthermore, the safe operation of the battery needs to be considered. Thus, the reward function needs to involve the battery’s physical constraints and energy cost, primarily composed of revenue from feeding PV and grid import costs. In this work, the reward function consists of two main components: the per-interval average cumulative cost resulting from the grid transactions and the reward on account of battery actions. First, the per-interval average cumulative cost replaces the energy cost in each interval [43]. It reduces the magnitude of reward values and expands the influence of the actions in the previous intervals [37]. Then, the function provides the battery operation with some soft constraints. When the battery has actions in the safe range of SOC, a reward of 1 is provided. Moreover, when the SOC violates the limits of the maximum and minimum values, a punishment of −20 is given. However, a punishment of −100 is also provided if the battery has no action because the agent may not run the batteries to avoid punishment. The values of punishment −100 and −20 are selected by several convergence experiments. The value of punishment can offset the cumulative positive reward so that the agent chooses the right actions in specific states [44]. The proposed reward function guides the agent in using on-site energy sources and reducing the purchased electricity from the grid. Additionally, it directs the agent to choose the actions in the safe range of SOC. The reward function can be expressed by Equations (12)–(14).

$$m_{pri}\left(t\right)=\frac{1}{T}{\displaystyle \sum _{t=1}^T{P}_{gr}\left(t\right)\times R_{grid}\left(t\right)\times \Delta t},$$

(12)

$$r_{ba}(t)=\left\{\begin{array}{l}1, \mathit{if} \mathit{action} \ne 0 \mathrm{and} {\mathit{SOC}}_{\mathit{min}} \le SOC\left(t\right) \le {\mathit{SOC}}_{\mathit{max}}\\ -100, \mathit{if} \mathit{action}=0 \mathrm{and} {\mathit{SOC}}_{\mathit{min}} \le SOC\left(t\right) \le {\mathit{SOC}}_{\mathit{max}}\\ -20, \mathit{if} SOC\left(t\right)>{\mathit{SOC}}_{\mathit{max}} \mathrm{or} SOC\left(t\right)<{ \mathit{SOC}}_{\mathit{min}}\end{array}\right.,$$

(13)

$$r(t)=-m_{pri}\left(t\right)\times a+r_{ba}\left(t\right)\times b,$$

(14)

where $m_{\mathit{pri}}\left(t\right)$ represents the cost from the grid transactions, which comprises revenue from PV sales and expenditure from grid purchases (Yen). $P_{\mathit{gr}}\left(t\right)$ denotes the power of buy or sell electricity (kW). If $P_{\mathit{gr}}\left(t\right)$ is positive, the system purchases electricity from the grid, and if it is negative, the system sells electricity to the grid. $r_{\mathit{ba}}\left(t\right)$ represents the reward and penalty the battery can receive solely by choosing its action, and r(t) represents the combined reward through performing the selected action. The reward weights, denoted by a and b, are set at 0.0001 and 1 in this study, respectively. The weights make the cumulative reward of all steps less than 0 and close to 0 indefinitely, which benefits the convergence of training models.

3.1.4. Parameter Settings

Table 2. Parameters in Q-learning, DQN, and DDPG.

Parameter	Q-Learning	DQN	DDPG
Discount factor γ	0.9	0.9	0.9
Learning rate α	0.001	0.001	0.001
ε-greedy initial value	0.9	0.9	0.9
ε-greedy decay factor	0.9	0.9	0.9
ε-greedy minimum value	0.01	0.01	0.01
Replay memory capacity	-	100,000	100,000
Batch size	-	32	32
Target net update steps	-	300	300
Number of hidden layers	-	2	2
Number of hidden units	-	(64, 64)	(64, 64)
Soft update coefficient	-	-	0.01

shows the values of parameters used in Q-learning, DQN, and DDPG agents. It is an important part of building RL models to achieve convergence and obtain optimal optimization results by adjusting hyperparameters continually. The settings of these parameters are chosen through continuous experiments and experiences. It is common to directly observe the trend of the reward convergence curve and refer to the existing literature. Other parameters keep default settings in the simulation environment. It is known that the proposed RL agents use the same values of parameters except for several special parameters.

3.2. Rule-Based Operation

The rule-based method involves formulating regulated rules based on extensive experience, providing an operational strategy for optimizing energy systems [16]. In this work, the rule-based operation mainly aims to control the charge/discharge power of a battery. It is known that the load peaks of ZEHs shift to the periods of 3:00–6:00 and 19:00–22:00. An amount of PV generation is surplus over the daytime. Thus, the control rules are as follows: First, the battery discharges accordingly with the load demand at 3:00–6:00 and 19:00–22:00. Additionally, PV energy is charged at 6:00–19:00.

3.3. Data Resource and Evaluation Method

In this work, we consider the energy system of a real ZEH located in Kyushu, Japan. We collected energy system data for the house over two distinct periods: from 1 January 2021 to 28 February 2022, and from 1 March 2018 to 28 August 2018. The collected data include load, PV generation, PV power sold, and power purchased from the grid, among other variables. Furthermore, the work uses the day-ahead market prices of Kyushu as real-time electricity prices with certain real-time and flexible features. The prices have been preprocessed, as shown in Figure 3. A 5-kWh battery with an 85% charge and discharge efficiency was used in the simulation. Additionally, the simulation incorporates a maximum of 1.5 kW charge power and a maximum of 1.25 kW discharge power and ensures that the SOC remains within the range of 20–95%. It is important to note that the simulation was conducted with a Python environment. Furthermore, the study was able to more accurately quantify the optimization and regulation of PV consumption by establishing quantitative relationships between distributed PV and user load. Several common assessment parameters were utilized; the details can be found in

Table 3. Common assessment parameters for evaluating the ZEH energy system.

Variables	Formula	Definition
Self-sufficiency ratio [14,45,46,47]	$\frac{{\displaystyle \sum P_{pv-\mathit{self-consumed}}\left(t\right)}}{{\displaystyle \sum P_{load}\left(t\right)}}\times 100\%$	The ratio of renewable power consumed by the user to total load demand.
Self-consumption ratio [14,45,46]	$\frac{{\displaystyle \sum P_{pv-\mathit{self-consumed}}\left(t\right)}}{{\displaystyle \sum P_{pv}\left(t\right)}}\times 100\%$	The ratio of renewable power consumed by the user to total renewable power.
Feed-in ratio [48,49]	$\frac{{\displaystyle \sum P_{pv}}\left(t\right)-{\displaystyle \sum P_{pv-\mathit{self-consumed}}\left(t\right)}}{{\displaystyle \sum P_{pv}\left(t\right)}}\times 100\%$	The ratio of renewable power sold to the grid to total renewable power.

.

4. Results and Discussions

4.1. Training Performance

The agents are trained using the dataset measured from 1 January 2021 to 31 December 2021. Then, the datasets measured from 1 January 2022 to 28 February 2022, and from 1 March 2018 to 31 July 2018 are used to test. The dataset is measured every half hour. The proposed Q-learning, DQN, and DDPG models are trained for 60 episodes to obtain the operation policy of the ZEH energy system. Each episode has 17,520 steps. It takes approximately 15 min per episode to finish the training of DQN and DDPG. The Q-learning agent takes 3 min per episode, offering a considerable amount of time savings. The optimization goal of the proposed agents is to minimize the operation cost of the ZEH energy system. Figure 4 illustrates the convergence of the cumulative operation cost in the training process. It can be seen that the proposed agents all keep learning and ultimately achieve a convergent policy. The training process goes through two stages, exploratory and convergence. Before 20 episodes, Q-learning and DDPG agents tend to explore actions, and the cumulative operation cost exhibits a reducing trend. After 20 episodes, Q-learning and DDPG agents achieve convergence. The DQN agent attains the convergence state and stabilizes at the highest value of operation cost after 10 episodes. Figure 4 shows that the cumulative operation costs decrease steadily before the convergence state in the training process. The Q-learning agent achieves the lowest cumulative operation cost at the end of the 60 episodes compared to the other proposed agents. The operation cost curve of the DQN agent displays evident fluctuations. The convergence reveals the proposed RL algorithm can obtain the available operation policy through continuous learning and exploratory. The training performance of the proposed RL agents also shows the effectiveness of the reward function.

4.2. Analysis of Testing Results

4.2.1. Effect of ZEH Energy System Scheduling

In this work, we analyzed the optimization of Q-learning, DQN, and DDPG agents compared to the rule-based operation. To explore and evaluate the optimization differences of the building energy system balance using the proposed RL agents, we analyzed two typical summer and winter weeks. The test results of energy system operation in a typical summer week are shown in Figure 5. Here, the battery importing is positive, and exporting is negative. For the grid exchange flows, the positive represents the system purchasing electricity from the grid, and the negative represents the system selling electricity to the grid. It is clear that electricity prices are at their highest during 10:00–14:00 when PV energy production is at its peak. The rule-based model uses this surplus PV electricity to charge for meeting user load demand at the specified times of the day. However, the models based on Q-learning, DQN, and DDPG demonstrated their flexibility by selling surplus PV electricity and discharging to sell to the grid according to price fluctuations. During 23:00–5:00, the models based on DQN and DDPG usually satisfied the load and charge from the grid when the real-time electricity prices were low. After 20:00, the proposed optimization models regulated battery discharging to satisfy the building demand as PV generation levels were near zero. When demand peaked during 3:00–5:00, rule-based battery charging or discharging followed the established rules. In this typical summer week, the DQN and the Q-learning agents showed similar learning effects. However, the DDPG agent performed well in learning the ZEH energy system scheduling policy. The work aimed to improve PV generation local consumption and realize the minimization of energy costs. The DDPG agent used PV power to charge with relatively high power before the electricity prices peaked. Additionally, it discharged timely for the users’ demand when electricity prices were relatively high.

The building energy system responses in a typical winter week are shown in Figure 6. It is known that PV generation decreases significantly in winter thanks to sunlight attenuation, and the load increases due to heating and hot water needs, etc. As a result, PV generation is insufficient to charge the batteries and meet the load, necessitating the purchase of additional electricity from the grid. The trends of grid transactions across all four models were consistent for the typical week. Real-time prices revealed that the tariff was lower during periods with high intra-day PV generation and increased when PV power was low or non-existent. In addition, the tariff for the first four days of this winter week was lower than that for the last three days. The results show that the building energy system charged continuously during 23:00–5:00 of the first four days despite PV generation’s zero output. Although the reward function considered giving a certain penalty to the proposed agents when the batteries do not work, these situations still arise in the energy system operation during winter periods.

It is noted that Q-learning and DQN typically select safer actions, rarely resulting in high-power charging or discharging in energy system operation. On the other hand, the DDPG model utilizes high-power actions more frequently. The battery in the model based on Q-learning tends to keep to low powers as actions, and its charging and discharge powers are greatly impacted by real-time electricity prices. This reveals the operation strategy based on the Q-learning algorithm obtains more flexibility. The proposed models obey the limits of the battery in their optimization. The agents learn the regularity of the ZEH energy system better. It is helpful for agents to judge if the operating environment stays within safe range though taking high-power actions and predicting the next steps. Thus, the DQN and Q-learning agents utilize relatively few high-power actions as they make decisions under their acquired strategies.

In addition, the paper highlights the effect of the proposed RL agents on load shifting. This work chose January, May, and July as winter, midseason, and typical summer months, respectively. Figure 7 shows the ZEH average net load curve using different optimization methods over three typical months. The net load is the electricity required to satisfy the demand not fulfilled by renewable energy, which is purchased from the grid by users. The net load peaks of this user can be observed during 2:00–6:00 and 19:00–23:00 over the three typical months. The highest net load peaks arose in operations based on Q-Learning among the proposed RL agents, and the agents all performed better than the rule-based model. In total, the building energy system model based on DDPG achieved the best peak-reducing results, effectively alleviating the electricity peak period load demands of users.

4.2.2. PV Self-Consumption

In the simulation environment of Python, the work presents the optimization model based on Q-learning, DQN, and DDPG. The optimization results of PV consumption in building energy system operation are shown in

Table 4. Operation results of optimization based on rule-based, Q-learning, DQN, and DDPG over the seven-month period tested.

Algorithm	Self-Consumption Ratio	SELF-Sufficiency Ratio	Feed-in Ratio
Rule-based	43.3%	32.2%	53.7%
Q-learning	47.4%	35.2%	48.7%
DQN	46.7%	34.7%	45.7%
DDPG	49.4%	36.7%	44.1%

. The system based on DDPG shows an outstanding self-consumption ratio of 49.4%, outperforming the compared rule-based result of 43.3%. The self-consumption levels with Q-learning and DQN methods are approximately the same at 47%. Compared to the self-consumption ratio of 35.8% without building energy system optimization, the proposed RL agents show a significant improvement in PV consumption ability. This parallel improvement in self-sufficiency level is mirrored in the increased PV self-consumption. More PV local consumption equates to fewer feed-in PV generation instances. The DDPG method shows a decrease of 9.6%, while the DQN and Q-learning methods reduce by 8.0% and 5.0%, respectively, compared to the rule-based method.

By assessing the self-consumption ratio and feed-in ratio in Table 4, the PV generation loss can be calculated accordingly. It is revealed that the model based on DQN incurs the highest PV loss, 7.6%, while the battery charging and discharging in the Q-learning model results in a PV loss of 3.9%. On the other hand, the rule-based model produces the smallest loss, 3.0%, in the electricity schedule. The 5 kWh battery capacity, in combination with the proposed RL algorithms and the rule-based method, serves to reach the maximum energy self-sufficiency ratio of 36.7% and PV self-consumption ratio of 49.4% in the ZEH. This presents that the ZEH energy system still has much potential to consume PV locally. As the battery capacity is limited, the local consumption level is primarily determined by the mismatch between the load demand generated by the operation of heat pumps, air conditioners, etc., and PV generation.

Table 5. Operation results of optimization based on rule-based, Q-learning, DQN, and DDPG in typical months.

	Typical Month	Winter	Midseason	Summer
Variables
Self-consumption ratio	Rule-based	73.0%	27.2%	44.2%
	Q-learning	79.2%	31.5%	46.4%
	DQN	78.1%	31.7%	47.4%
	DDPG	80.0%	34.0%	48.6%
Self-sufficiency ratio	Rule-based	19.9%	44.3%	56.8%
	Q-learning	21.6%	51.3%	59.6%
	DQN	21.3%	51.7%	60.9%
	DDPG	21.8%	55.5%	62.4%
Feed-in ratio	Rule-based	26.2%	69.4%	51.7%
	Q-learning	15.9%	64.2%	48.9%
	DQN	12.5%	61.2%	46.4%
	DDPG	10.1%	60.3%	45.8%

shows the PV consumption of the ZEH energy system in typical winter, summer, and spring-to-summer months. It can be seen that more than 70% of PV electricity in winter is used locally. Consequently, increasing PV electricity feed-in in the grid challenges its stability. The model based on DDPG is found to have the highest self-consumption ratio in typical months, even reaching 80% in the typical winter month, while the model based on DQN outperforms Q-learning.

4.2.3. Energy Cost Saving

Table 6. Cumulative operation cost in typical months.

	Typical Month	Winter	Midseason	Summer	Total Cost
Operation Cost (Yen)
Rule-based		23,076	−3053	−2888	43,301
Q-learning		22,749	−2995	−3712	41,374
DQN		22,616	−2987	−3715	40,868
DDPG		22,528	−3040	−4201	40,201

summarizes the cumulative operation cost resulting from simulating the operation of the ZEH energy system with test data. Negative values represent the revenue obtained by the users, while positive values represent expenditures due to the electricity schedule. Firstly, when using the rule-based results as a benchmark for comparison, the model based on DDPG enables an economization of 7.2% in the operation cost of the ZEH energy system over the seven-month period tested. Additionally, the operation cost based on the DQN algorithm is found to economize 1.1% more than the Q-learning algorithm. Furthermore, the proposed models are all able to obtain relatively high revenue in the operation of a summer month among the typical months. It is worth noting that the revenue based on DDPG is 45.5% higher in the typical summer month. Moreover, the transaction between the grid and battery is significantly increased due to the decreasing PV electricity and the rising demand. These charging and discharging actions result in a loss, further increasing the operation cost, which explains why the operation cost based on DDPG only saves 7.2%.

5. Conclusions

This work applied RL algorithms, including Q-learning, DQN, and DDPG, to optimize the operations of the hybrid energy system in ZEHs. The optimization goal was formulated to minimize the energy operation cost in the real-time electricity market while improving renewable electricity self-consumption. The reward function was properly designed to guide the learning process of RL agents. The Q-learning, DQN, and DDPG agents were trained and tested based on the measured data. Results compare the dispatch performances of the proposed RL agents in various situations in terms of dynamic PV self-consumption and energy cost of test periods. The main conclusions can be drawn as follows:

The proposed RL algorithms showed good performances in achieving the supply and demand balance of building energy systems with uncertainty in various situations. The DDPG agent showed the best learning results, which can control the battery by taking high-power actions in typical weeks. The performances of the agents proved the effectiveness of the designed reward function. In detail, the optimization based on the DDPG algorithm outperformed in shifting load in terms of cost saving. Additionally, the Q-learning agent showed more flexible action control of the battery with the fluctuation of real-time electricity prices than the DQN algorithm.

The proposed RL algorithms increased the local PV electricity consumption by managing the behavior of a 5 kWh battery. Compared with the rule-based schedule results, the DDPG agent achieved the highest self-consumption ratio of 49.4% and self-sufficiency ratio of 36.7% in the testing results. The DQN agent performed better than the Q-learning agent in improving PV electricity consumption. However, the Q-learning agent achieved a lower energy loss of 3.9% caused by the charging/discharging of the battery during the test periods.

The DDPG agent achieved the lowest energy cost over the seven-month period tested. The energy cost saving was 7.2% higher than rule-based control, realizing better economic performance. Then, the energy cost based on DQN showed better economic benefits than Q-learning control. Compared with the rule-based testing results, the revenue of ZEH energy system operation optimization based on DDPG was more than 45.5% in a typical summer month.

Future research may expand on our findings to improve the evaluation system considering energy flexibility and energy efficiency and use multi-agent algorithms for the optimization operation of ZEHs. Moreover, other storage, such as thermal storage and hydrogen storage, can be considered to cooperate with the grid-building interaction to increase the potential of renewable energy self-consumption in buildings.