1. Introduction
Renewable energy, such as wind and solar energy, is essential for the energy decarbonization [
1]. Microgrid is an important form for renewable energy integration to the power systems [
2]. Hydrogen energy is another type of clean and low-carbon energy. The combustion product of hydrogen is water with zero-carbon emissions [
3]. For microgrid systems with high renewable energy integration, hydrogen energy can be used as a long-term energy storage to improve the utilization of renewable energy and reduce carbon emissions. The renewable energy is intermittent and random, and brings great challenges to the operation of the microgrids [
4].
To address the economic dispatch problem in microgrids containing hydrogen storage, a mixed integer nonlinear dispatch model for a microgrid with 100% renewable energy generation is proposed in [
5], and the GAMS solver is used to optimize the operation strategy of hydrogen storage and improve the economic efficiency of the microgrid in the day-ahead market. In [
6], a nonlinear scheduling model for a microgrid containing fuel cell and hydrogen storage systems is proposed and the CONOPT solver is used to optimize the energy purchase cost of the microgrid. In [
7], an optimization model to schedule an islanded microgrid with various resources, including photovoltaic generation and hydrogen energy system, is proposed. The problem is represented as a mixed integer linear program problem and solved by CPLEX. In [
8], the retail price problem of the electricity energy retailer that owns plug-in electric vehicles and hydrogen storage systems is proposed. The proposed model is verified by simulation using GAMS. In [
9], the harmony search algorithm is used to optimize the hydrogen production capacity of the hydrogen storage in the microgrid to reduce the operating cost. In [
10], a hybrid AC-DC microgrid model containing electric vehicles and hydrogen fuel cells is presented, and the operating scheme is optimized using an improved teacher learning algorithm. In [
11], the genetic algorithm is used to optimize the life cycle cost of the microgrid containing hybrid electric-hydrogen energy storage. In [
12], the particle swarm algorithm is used to solve the multi-objective energy management problem of renewable energy microgrid containing electric-hydrogen hybrid energy storage to improve the system efficiency.
The conventional mathematical programing algorithms in the above literature are computationally efficient. However, these methods tend to be trapped in local optima when the problem is nonlinear and nonconvex. The heuristic algorithms have better global optimization capability, but suffer from slow convergence and poor generalization. In addition, the above literature mainly focuses on the day-ahead scheduling problem of microgrid, and relies on the accurate predictions of renewable energy and load.
Deep reinforcement learning is a machine learning method with the ability to perceive the environment and address uncertainties. Currently, deep reinforcement learning has been used to achieve certain results in several areas, such as reactive power optimization [
13,
14], electric vehicles [
15,
16], and power markets [
17,
18]. In terms of optimal operation, the deep reinforcement learning algorithms is used in [
19] to solve the energy management problem of residential energy system with electricity, heat, and gas demand. In [
20], a microgrid scheduling model is proposed and deep reinforcement learning algorithms is adopted to reduce the power purchase cost. However, this literature fail to consider the impact of hydrogen energy storage system on the microgrid operation. In [
21], a coordinated control method for electrochemical and hydrogen energy storage in microgrid based on deep reinforcement learning is proposed. However, the hydrogen storage model is simple and ignores the electrolyzer efficiency characteristics, which has significant influence on the operation of microgrid. Moreover, only sub-optimal solution can be found because of the discretization of the action space.
In this paper, an optimal operation model for a microgrid with hydrogen energy storage system is developed. The efficiency-power model of the electrolyzer is established based on linear interpolation to evaluate the operating cost of the electrolyzer. The objective of the optimal operation model is to reduce the operation cost and guarantee the safety of the system. The deep deterministic policy gradient (DDPG) algorithm is used to optimize the operation scheme of the microgrid. The DDPG algorithm can deal with the continuous action space problem and obtains better operation scheme compared with the conventional algorithms. Additionally, the trained DDPG model is used in new scenarios. The simulation results show that the DDPG algorithm has generalization capabilities.
The main contributions can be summarized as:
A refined model represents the electrolyzer efficiency characteristics based on the linear interpolation method is proposed;
An optimal operation model for a microgrid with hydrogen storage is proposed. The electrolyzer efficiency characteristics model is incorporated into the optimal operation model;
The DDPG algorithm is adopted to solve the optimal operation model, which has a continuous action space.
2. Model of the Microgrid System
A microgrid can increase the integration of renewable energy and reduce the carbon emissions of the whole energy system. In this paper, an islanded microgrid was constructed. The structure of the microgrid is shown in
Figure 1. The microgrid included the load, a microturbine, a photovoltaic (PV) generation device, a battery energy storage system (BESS), and a hydrogen storage system. The hydrogen storage system consisted of an electrolyzer, a hydrogen storage tank, and a solid oxide fuel cell (SOFC). The hydrogen storage system [
22] can provide regulation capability to the microgrid and improve the system reliability.
2.1. Electrolyzer Efficiency
Electrolyzer efficiency
is the efficiency of the hydrolysis reaction at constant temperature and pressure. The electrolyzer efficiency [
23] consists of voltage efficiency
and current efficiency
as below:
The current efficiency, also known as Faraday efficiency, can be expressed as:
where
I is the stack current of the electrolyzer.
Voltage efficiency is the ratio between the theoretical decomposition voltage of water and the actual decomposition voltage, which can be expressed as
where
is the theoretical decomposition voltage, which is generally 1.482 V;
is the actual decomposition voltage. Under the pressure
of 1.01 × 10
5 Pa,
depends on the unit current density during the electrolysis of water, as below:
where
is the unit current density;
T is the working temperature of the electrolyzer;
is the reversible voltage of the electrolytic water;
is the voltage drop caused by the resistance of electrolyte;
and
are the hydrogen overpotential and oxygen overpotential generated by the electrolytic water, respectively.
,
,
and
are determined by
where
is the resistance of the electrolyte;
R is the universal gas constant,
F is the Faraday constant;
and
are the charge transfer coefficients of anode and cathode, respectively;
and
are the exchange current densities of anode and cathode, respectively;
and
are the electron transfer numbers of anode and cathode, respectively. The input power of the electrolyzer
is related to the electrolyzer current
as follows
The relation between the input power and the electrolyzer efficiency can be obtained by Equations (1)–(9). However, the relation is complicated and contains logarithmic calculations. Thus, it is difficult to find the corresponding electrolyzer efficiency based on the input power of the electrolyzer in the microgrid scheduling problem.
In order to simplify the electrolyzer efficiency characteristics model, this paper firstly obtained
and the corresponding
according to
j. Then, the electrolyzer efficiency characteristic curve was obtained based on
and
, as shown in
Figure 2. Twenty points on the efficiency characteristic curve were taken as the original data to form the data table. When solving the scheduling problem, the electrolytic cell efficiency corresponding to
can be quickly found by looking up the table and linear interpolation, as shown below:
where
and
are the two power values nearest to
in the data table;
and
are the corresponding electrolyzer efficiencies of
and
in the data table.
When the input power and the efficiency of the electrolyzer are determined, the hydrogen production power of the electrolyzer can be calculated according to Equation (11):
where
is the hydrogen production power of the electrolyzer.
Different from the conventional fixed efficiency model of electrolyzer, the hydrogen production power was obtained by multiplying the power consumption of electrolyzer and the respective efficiency obtained from the electrolyzer efficiency characteristic model.
2.2. Economic Dispatch Model of Microgrid
2.2.1. Objective Function
The total cost
in all scheduling periods of a day is set as the objective function. This objective function not only covers the economic benefits of microgrid, but also takes into account the environmental benefits of microgrid, as below:
where
is the whole dispatching cycle;
t is the time step, and the scheduling interval is 1 h;
is the operating cost of the microturbine at time
t;
is the CO
2 emission cost of the microturbine at time
t;
,
and
are the operation costs of the BESS, electrolyzer, and fuel cell, respectively. The above operation costs can be determined by
where
,
, and
are the power generation cost coefficients of microturbine;
is the scheduling interval;
,
, and
are the operation and maintenance cost coefficients of BESS, electrolyzer, and fuel cell, respectively;
is the CO
2 emission coefficient of microturbine;
is the carbon emission price of carbon trading market;
is the power generation of microturbine at time
t;
is the charging or discharging power of BESS at time
t, and a positive value of
means the BESS is charged. Otherwise, BESS is discharged;
and
are the input power of electrolyzer and output power of fuel cell at time
t, respectively.
2.2.2. Constraints
Generally, in order to ensure the overall working efficiency of the hydrogen storage system, the electrolyzer and fuel cell cannot work at the same time. Therefore, the input power of the electrolyzer is regarded as the charging power of the whole hydrogen storage system, and the discharging power of the fuel cell is regarded as the discharging power of the whole hydrogen storage system, as below:
where
is the charging/discharging power of the hydrogen storage system at time
t, and a positive value of
means the hydrogen storage system is charged. Otherwise, the hydrogen storage system is discharged.
In addition to economic efficiency, the operation safety of microgrid also needs to be guaranteed. The operation constraints of microgrid are as follows:
The microgrid in this study is off grid. The power balance of the microgrid mainly relies on the output power of PV generation and microturbine. The imbalance power is regulated by BESS and hydrogen storage system. The power balance equation is
where
,
,
and
are the available PV generation, curtailment of PV generation, load power, and curtailment of load at time
t, respectively.
- 2.
Operating power constraints
To ensure the safety of the devices in microgrid, the operating power constraints are as below:
where
,
,
and
are the upper power limits of microturbine, BESS, electrolyzer, and fuel cell, respectively;
,
,
and
are the lower power limits of microturbine, BESS, electrolyzer, and fuel cell, respectively.
- 3.
Energy storage capacity
In order to avoid overcharging and over-discharging of energy storage, the states of charge (SOCs) of energy storage can be constrained as:
where
is the SOC of BESS at time
t;
and
are the upper and lower limits of SOC of BESS, respectively;
is the SOC of hydrogen storage system at time
t;
and
are the upper and lower limits of the SOC of hydrogen storage system.
The SOCs of the two energy storage devices can be calculated by the following equations:
where
and
are the charging and discharging efficiencies of BESS, respectively;
and
are the efficiencies of electrolyzer and fuel cell, respectively;
and
are the capacities of BESS and hydrogen storage tank, respectively.
Because the operating cost of microturbine is a quadratic function, the objective function is nonlinear. All of the constraints are linear. Thus, the whole model is a quadratic programing model that is nonlinear.
6. Conclusions
This paper proposes a refined model to represent the electrolyzer efficiency characteristics using the linear interpolation method. The electrolyzer efficiency characteristic model is combined with the model of the microgrid with hydrogen storage. Additionally, an optimal operation method based on the DDPG algorithm is proposed for the microgrid. According to the simulation results, the following conclusions can be drawn:
The electrolyzer efficiency characteristics model using linear interpolation method can describe the operation of electrolyzer more accurately. The proposed optimal operation method for the microgrid considering electrolyzer efficiency characteristics can reduce the PV curtailment and reduce the microgrid operation cost;
The optimal microgrid operation method based on DDPG algorithm can effectively reduce the operation cost and improve the microgrid efficiency compared with the method based on traditional algorithms, such as the GA and interior point method;
The optimal microgrid operation method based on DDPG algorithm has a certain generalization and can be used in in different scenarios.
However, the uncertainties of PV and load are not considered in this research, and the fuel cell efficiency is ignored. Future work will focus on the microgrid operation optimization strategy under uncertain environments and take into account the characteristics of fuel cell to make the operation model more realistic.