A Hawkes Model Approach to Modeling Price Spikes in the Japanese Electricity Market

Abstract

The Japan Electric Power Exchange (JEPX) provides a platform for the trading of electric energy in a manner similar to more traditional financial markets. As the number of market agents increase, there is an increasing need for effective price-forecasting models. Electricity price data are observed to exhibit periods of relatively stable, i.e., low-magnitude, low-variance prices interspersed with periods of higher prices accompanied by larger uncertainty. The price data time series therefore exhibits a temporal non-stationarity characteristic that is difficult to capture with typical time series modeling frameworks. In this paper, we implement models for the occurrence of price spike events where spikes are defined as observing prices above a predefined threshold set here at 25 JPY/kWh. This value corresponds to about the 90th percentile of observed prices during peak trading periods. The price spikes time series is observed to be rare events that occur in clusters. We therefore propose to model the data as a Hawkes process whereby the occurrence of a spike event increases the probability of observing more spikes in the period immediately following a price spike event. We test two variations of the classical Hawkes model: the first variation models the change in the magnitude of the underlying intensity as a function of the magnitude of the price spike while the second variation models the change in the decay rate of the underlying intensity as a function of the magnitude of the price spike. An analysis of the performance of the models based on the mean absolute error (MAE) of the spike occurrence probability, a weighted accuracy index, and the Matthews correlation coefficient (MCC) metrics shows the effectiveness of the variable magnitude variation of the Hawkes model in generating short-term forecasts of the occurrence of price spike events. The modified Hawkes model especially outperforms other candidate models as the length of the forecasting horizon increases.

1. Introduction

1.1. Background

The liberalization of the electric energy sub-sector in many countries has allowed for the electricity to be traded in financial markets in a manner similar to other commodity markets such as company stocks [1,2,3]. The day-ahead electricity market such as the Japanese Electric Power Exchange (JEPX) [4] provides a mechanism for power suppliers and customers to trade electricity in an environment where the price is dictated by the market forces of supply and demand. These markets were mainly introduced with the goal of improving service quality and curtailing monopolistic tendencies of the original regional or national utilities [4]. Trading occurs under the supervision of an independent market operator (MO) who works closely with the actual system operator (SO) who keeps the original role of ensuring high levels of system reliability [5]. The primary goal for market participants is to operate efficiently and economically in the electricity market, which requires the design of proper strategies and tools based on power system requirements [6].

Prices in deregulated electricity markets are highly variable due to the dependence on several factors both on the energy supply and demand sides [7]. Several factors including weather conditions, fuel costs, power plant operating costs, and regulations contribute to the increasing price uncertainty in the market [8]. On the supply side, the cost of fuel to run thermal generators is typically dependent on oil prices which is quite variable in itself. Furthermore, an increase in power supplied by solar- and wind-based renewable energy generators escalates the uncertainty in the availability of power since these sources depend on relatively uncertain primary sources of energy [9]. The management of these non-dispatchable energy resources introduces new significant challenges in the context of a competitive market environment [10]. However, the largest source of uncertainty is probably the strategies adopted by various companies during the bidding process as they jostle for a slice of the market share. On the consumer side, well-documented and relatively predictable variations in power demand have an effect on prices [11]. Furthermore, in a two-sided market such as the JEPX, large power consumers or resellers participate in the bidding process with the objective of driving down prices to reduce their electricity bills. Certain aspects of the physical system such as limits in power that can be sent though certain power lines also has an effect on prices. These uncertainties mean that electricity prices are characterized by large spikes and dips, which has an effect on the financial benefits that suppliers and consumers can gain from the market. The first step in mitigating against the volatility in the market is to set up an accurate price-forecasting system. The obtained forecasts form an important basis for decision making by investors in the electricity market [12].

As with other commodity markets, electricity markets also experience periods of market shocks—a situation in which prices are driven much higher that normal due to extreme market conditions [13]. Price spikes can be the results of various issues in the market, including power plant and system failures or extreme weather conditions that drive up demand [14]. Such a scenario was observed in the JEPX around late December 2020/early January 2021 where prices hit a high of 220 JPY/kWh which is 25–30 times the average price of around 7–8 JPY/kWh. While the end consumers may not have felt the effects of these spikes since most are in fixed tariff plans, a similar occurrence in the Texas’ market ERCOT in the United States left households whose typical bills are 100 to 200 dollars with bills to the tune of tens of thousands of dollars. This further reinforces the need for mathematical models of the system that may be used to understand the underlying dynamics that lead to such extreme events.

1.2. Literature Review

1.2.1. Modeling of Electricity Prices

An overview of existing literature shows that numerous approaches have been proposed for the modeling and forecasting of short-term prices in electricity markets. References [15,16] give very good reviews on the topic. Similarly, a review specifically on probabilistic forecasting can be found in [17].

In [18], the authors provided a method for predicting next-day electricity prices using the autoregressive integrated moving average (ARIMA) methodology. A detailed analysis of the electricity prices time series using the ARIMA models preceded model fitting and an analysis based on the mainland Spain and Californian markets. Classical autoregressive integrated moving average (ARIMA) models with various AR and MA orders were also tested in [19]. Simulation results using data from the UK electricity market selected the ARIMA (4,1,2) model as the best and was found to achieve better results than persistence or a typical artificial neural network (ANN) model. A seasonal autoregressive integrated moving average model with exogenous variables (SARIMAX) for modeling electricity prices was proposed in [20]. The model was chosen so as to capture the seasonal variations of electricity prices. In [21], electricity price behavior in the Nordic electric power market was forecasted using both the Markov switching generalized autoregressive conditional heteroscedasticity (MS-GARCH) model and a set of different volatility models. The GARCH models aim to model the conditional volatility of the price time series. The study demonstrated that electricity price volatility was not only highly volatile but also strongly regime-dependent.

Given the highly nonlinear nature of the electricity price time series data, several authors have presented approaches based on newer machine learning models. An exhaustive analysis of existing forecasting models was carried out in [22], where 27 common approaches were analyzed with a conclusion that generally, deep learning models outperformed statistical models. They also concluded that hybrid models did not outperform simpler counterparts. A recurrent neural network (RNN)-based Model was proposed in [23] where multilayer gated recurrent units were proposed for electricity price forecasting. The use of dynamic trees for both very short and short term electricity price forecasting and the improvement in forecasting performance when compared to a typical random forest approach was illustrated in [24] with tests on the Iberian market. Reference [25] employed an artificial neural network (ANN) model with a focus on the selection and preparation of fundamental data that had noticeable impact on electricity prices.

In [26], a hybrid modeling approach that combined the features and strengths of the autoregressive fractionally integrated moving average (ARFIMA) model and the least-squares support vector machine (SVM) model was proposed. Similarly, a long short-term memory (LSTM) deep neural network combined with feature selection algorithms for electricity price prediction under the consideration of market coupling was proposed in [27]. An outlier-robust hybrid model for forecasting electricity prices that combined a simple outlier-robust extreme learning machine model and several other algorithms was developed in [28]. Their experiments based on electricity price data from the Australia and Singapore markets demonstrated the effectiveness of the model especially in dealing with the complex nonlinear characteristics and numerous outliers present in the price data.

1.2.2. Electricity Price Spikes Modeling

While there has been a lot of focus on the modeling and forecasting of electricity prices, there is less literature on modeling price spikes, i.e., the occurrence of extreme prices which has a significant effect on the operation of market agents. An economic analysis of price spikes was presented in [13] where the authors investigated the factors and mechanisms determining spikes in the Italian electricity market. Based on the market data, they performed a specific analysis of the auctions mechanisms and of the hourly bids and offers of electricity characteristics to determine how and why price spikes occurred. Their results showed that rigidity, which characterized both the demand curve and part of the supply curve, was the fundamental determinant of prices pikes. Fluctuations in renewable energy sources (RES) production also proved to be fundamental.

In [29], a stochastic regime-switching model with time-varying parameters was shown to capture the type of volatile price behavior observed in many deregulated spot markets for electricity. The mean prices in two price regimes and the transition probabilities were specified as functions of the offered reserve margin and the system load. The high-price regime corresponded to the observed price spikes that typically occurred during the summer months. In addition, the structure of the model was consistent with the actual hockey stick shape of the offers submitted by suppliers into the PJM market. Most capacity is offered at relatively low prices, and a few units are offered at much higher prices up to the price cap.

Dramatic rises in electricity prices can be observed during periods of market stress as highlighted in [30]. The authors treated abnormal episodes or price spikes as count events and proposed to build a model of the spiking process. The importance of persistence in the spiking process and its significance in building an effective model was highlighted. They adapted a Poisson autoregressive framework for integer-valued time series that accounted for the number of simultaneous stresses remaining latent and provided a model that could be estimated by maximum likelihood. The arrival and survival rates of price spikes were found to be dependent upon extreme temperature events and peak load. However, the model’s ability to capture the intrinsic persistence in price spikes was cited as more significant.

The forecasting of extreme price events, the occurrence of which was treated as a realization of a discrete-time point process, was the focus of [31]. An autoregressive conditional hazard (ACH) framework was used to analyze the drivers of the process and to forecast the probability of extreme price events occurring in real time. Abnormal loads were found to have a significant impact on the probability of a price spike and on the severity of the spike. Importantly, stochastic factors capturing the history of the process were found to be significant in explaining the occurrence of extreme price events.

An argument that there is increasing empirical evidence of increased price volatility and spikes in electricity markets as a result of fluctuating renewable energy production, extreme weather events, and other factors was presented in [32]. While price spikes are necessary to cover the fixed costs of power plants, they can also indicate market imperfections and anticompetitive behavior. Regulators have set market price caps to protect consumers and prevent abusive behavior by vendors. Additionally, some regulators have imposed temporary price caps during or after major events. In weak institutional frameworks, however, these ceilings may be driven by political motives rather than economic logic.

1.3. Study Contributions

This paper focuses on modeling processes that captures the underlying dynamics of deregulated electricity markets. Such models could be used by market agents for forecasting purposes and hence developing bidding strategies or may be used by market operators and regulators to detect extreme circumstances. While there is much literature on the modeling and forecasting of electricity prices time series, research dealing specifically with the extreme price events is significantly scarcer. However, recent events have shown that these extreme prices have the biggest impacts on market players and there have been several cases of electricity retailers going bankrupt after a short run of price spikes. Unlike most papers on the modeling of prices in electricity markets, this study focuses specifically on modeling price spikes by disintegrating spikes from “normal” periods rather than modeling the electricity market prices in their entirety. Available papers take an aggregate approach when modeling price spikes—modeling the number of events in a day—thereby losing information on the time period in which the spike would occur. Given that extreme prices seem to coincide with high-demand periods, it is important to provide models that specifically isolates the event occurrence’s time periods. In addition, while the existing literature typically generate one-step-ahead forecasts, here we provide a method for generating forecasts a few days ahead (our tests generate 14-day-ahead spike occurrence probability forecasts).

The price spikes time series are modeled using the Hawkes model which is typically used to model nonstationary point processes. We present results that demonstrate the effectiveness of a modified form of the Hawkes model [33] in the short-term forecasting of the occurrence of price spike events. Modeling is done at half-hourly time resolutions as opposed to average day-ahead prices since prices vary throughout the day depending on the time of day. No assumptions or simplification are made on electricity prices either, so weekend prices are not ignored. We present modifications on the classical Hawkes model showing the effect of including spike magnitudes information on the spike event occurrence’s forecasting performance. Finally, we present simulations on the Japan Electric Power Exchange on which there are very few studies.

1.4. Paper Organization

The remainder of this paper is organized as follows: Section 2 gives an introduction to the Japanese electricity market including the characteristics of the electricity grid, a look at the prices datasets, and the definition of price spikes. Section 3 presents the modeling approach introducing the proposed models, the parameter extraction approach, and the generation of short-term price spikes forecasts. Simulation results based on data from the JEPX are presented and discussed in Section 4, and the study conclusions are drawn in Section 5.

2. The Japanese Electricity Market

2.1. Introduction to the Spot Market and the Power Grid

Following the trend toward deregulation in the electric power industry in Western countries, the liberalization of the electricity generation sector started in 1995 in Japan, followed by retail supply liberalization for customers receiving extrahigh voltage (20 kV or above) in 2000 [34]. The scope of deregulation was expanded in different stages afterwards. However, power shortages and other issues caused by the 2011 Great East Japan Earthquake prompted discussion about the ideal structure of the country’s electric power system and its reform. Based on this, full liberalization of the Japanese electricity market was achieved in 2016. The Japan Electric Power Exchange (JEPX) was established in November 2003 and began trading in April 2005. The purpose of the JEPX is to handle electricity transactions on the exchange. This research focuses on forecasting prices spikes in the JEPX day-ahead market.

The general structure of the Japanese electricity market follows closely those of more established markets such as the PJM (Pennsylvania, Jersey, and Maryland) Power Pool interconnection in the US [35] and the European Nord Pool covering the Northern European countries, such as Norway, Sweden, Denmark, and Germany [36]. While most of the energy is traded in the day-ahead market, there is an intraday market for settlement of hour-ahead forecasted load demand. However, unlike the markets in the United States, a real-time market [37] is yet to be implemented. Market clearing is carried out on a 30 min time resolution unlike the Italian or Spanish markets where clearing is done on a 1 h time resolution. The Japanese market also uses a zonal marginal pricing policy with nine trading areas in a structure similar to most European markets but unlike the PJM, which uses a locational marginal pricing (LMP) policy [38]. Similar to the Swedish, Spanish, Italian, and most European markets, the market operator (MO) in the Japanese market is separate from the system operator (SO) but unlike the Australian, PJM, and UK markets where market and system operation functions are carried out by the same entity. Apart from the market structure, the Japanese system is unique in having two system frequencies (50 Hz in the east and 60 Hz in the west) [4] within the same market, sometimes leading to significant differences in prices within the market. In addition, there is no international connection unlike the highly connected European markets which means that localized mismatches in supply and demand cannot be offset by imports or exports from or to nearby grids.

There are 10 operational areas (9 in the main island of Honshu and the Okinawa area that covers the southern islands) as shown in Figure 1 [34]. These areas correspond to the regions originally operated by the main power utilities before system deregulation and are currently operated by separate system operators. The JEPX handles transactions for the nine main areas on a thirty-minute time resolution resulting in 48 trading periods per area per day on the day-ahead electricity market.

The transmission capacity is limited in the connections between areas. These limitations result in transmission congestion, hence differences in prices between areas. Particularly the HVDC interconnection linking the Tokyo and Chubu areas result in differences between prices in the eastern grid operated at 50 Hz and western grid operated at 60 Hz. The overall structure of the electricity market is as shown in Figure 2 and the main market participants are the electricity generation companies and electricity retailers involved in wholesale power transactions. The number of participating generators in the market are 986 and a total of 730 retailers as of September 2022 [39].

2.2. JEPX Dataset

The JEPX electricity price and traded energy volumes data are publicly available on the exchange’s website [40]. The available data include traded volumes and corresponding system and area market clearing prices (MCPs) in 30 min resolutions giving 48 commodities per area, per day.

Table 1. Summary statistics of area prices (JPY/kWh) in the JEPX for the period 1 April 2016 to 31 March 2022.

Area	Mean	Median	Std. Dev	Skewness	95th Percentile
Hokkaido	12.72	10.54	12.01	8.90	25.67
Tohoku	10.82	8.67	11.75	9.83	23.00
Tokyo	10.93	8.71	11.85	9.67	23.37
Chubu	9.91	7.66	10.96	9.48	21.87
Hokuriku	9.89	7.66	10.94	9.53	21.54
Kansai	9.88	7.66	10.93	9.57	21.24
Chugoku	9.87	7.65	10.93	9.57	21.24
Shikoku	9.86	7.65	10.95	9.61	21.22
Kyushu	9.16	7.36	10.69	10.21	19.45

shows the summary statistics of the MCP data for each of the nine areas in the JEPX for the period spanning 1 April 2016 to 31 March 2022. The values clearly indicate differences in the grid with prices in the eastern areas of Hokkaido, Tohoku, and Tokyo being generally higher than prices in the western grid. In addition, prices in the northern island of Hokkaido are generally highest while prices in the southwestern region of Kyushu being the lowest. Prices in the western regions of Chubu, Hokuriku, Kansai, Chugoku, and Shikoku are very similar indicating adequate transmission capacity between these areas leading to almost always a single MCP between them. Table 1 also shows the 95th percentile value of the prices time series. These values give a sense of what would be considered as extreme prices in the market.

A fundamental analysis of the JEPX spot-market time series data shows that prices have a clear spatial and temporal dependence. This dependence is illustrated in Figure 3, which shows the average area MCPs (calculated on a thirty-minute resolution) for the period spanning 1 April 2016 to 31 March 2022 covering the first six years since the start of full market liberalization that allowed for the competitive retail of electricity to individual consumers. The data are grouped to show the average prices for both workdays and nonworkdays (weekends and holidays).

On the spatial axis, prices are seen to be different for different areas. Generally, prices are observed to be higher in the eastern regions of Hokkaido, Tohoku, and Tokyo and lower on the western side. The southwestern region of Kyushu exhibits especially relatively lower prices. This dependence is explained by constraints on the amount of power transferable through the interconnections between regions. Capacity limits of the HVDC interconnection linking the Tokyo and Chubu areas especially results in significant differences between prices in the eastern grid operated at a frequency of 50 Hz and the western grid operated at 60 Hz.

Temporally, MCPs are dependent on the type of day, i.e., workday or non-workday, and the time of day, i.e., there are clear peak and off-peak periods within a day. This temporal dependence is explained by the socioeconomic behavior of electricity users with morning and evening peaks. Reduced energy-intensive activities on weekends and holidays result in generally reduced demand and consequently lower prices during these days, as compared to typical workdays.

Considering the above characteristics of the JEPX price data, we propose the use of individual models for each area and each trading time period to handle the spatial dependence and time-of-use dependence, respectively. In addition, we propose a transformation approach to handle the type-of-day dependence. The difference in prices due to the type of day is handled by transforming the weekend/holiday prices ${y_d^{a,t}}^{\prime }$ to equivalent weekday prices $y_d^{a,t}$ using scaling factors $k_d^{a,t}$ as:

$$y_d^{a,t}=k_d^{a,t}\times {y_d^{a,t}}^{\prime }$$

(1)

where $k_d^{a,t}$ is the ratio of the average workday price to the average non-workday price up to day d for area a and trading period t.

Figure 4 shows the effect of the price transformation technique on the empirical cumulative distribution curves (cdfs) of the MCPs for the Tokyo area. In the original form of Figure 4a, the MCPs for workdays and non-workdays can be thought of as belonging to different probability distributions which would require a regime-switching-type model to properly handle this characteristic. However, as seen from Figure 4b, the transformation of Equation (1) “shifts” the distribution of the non-workday prices to be similar to that of the workday prices allowing for the fitting of a single model and avoiding the breaking of the time series to workday and non-workday portions. We then fit the models on the transformed time series and carry out the inverse of the above transformation when determining the magnitude of the forecasted price spikes.

2.3. Definition of Price Spikes

Extreme prices in electricity markets can have devastating impacts on market participants even if they last for just a short period of time. Cases of electricity resellers that have been driven to bankruptcy due to these “price spikes” have been observed in the last few years even in more established markets in the US, UK, and Europe. In an environment of tight margins, even with a conservative hedging strategy, one can be left exposed to extreme price risks especially since spikes tend to occur during peak demand periods where retailers will be typically underhedged.

We define a price spike on day d and in area a and trading period t, $u_d^{a,t}$, as an observed price value $y_d^{a,t}$ above a predefined threshold $y^{\delta }$. Mathematically, this is represented as:

$$u_d^{a,t}=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}{y}_d^{a,t}\le y^{\delta }\hfill \\ 1,\hfill & \mathrm{otherwise}\hfill \end{array}\right..$$

(2)

For the electricity retailer procuring energy from the spot market, this price threshold would define a risk value above which the potential of loss becomes significant.

Figure 5 shows the price ranges (10th to 90th percentile) of area spot prices in the JEPX for the data spanning 1 April 2016 to 31 March 2022. Here, we only show four representative areas since the price characteristics in several areas are quite similar as seen from the data in Table 1 and Figure 3. Apart from the Hokkaido area in which prices were a little higher, the peak of the 90th percentile curve was about 25 JPY/kWh, and prices above this value could be considered extreme. In fact, in the entire dataset, only 3.7% of the area prices were greater than this threshold. We therefore adopted a value of 25 JPY/kWh as a consistent price spike threshold throughout this paper, though the analysis and results would be similar with any other reasonable threshold value.

The series of data points showing prices above the threshold value is the price spikes time series. This time series has two components: (1) the occurrence or non occurrence of a spike and (2) the magnitude of a spike should it occur. The temporal evolution of price spikes for different trading areas and selected trading periods is illustrated in the plots of Figure 6. Each price spikes time series is presented as two plots: the barcode plot (top plot) illustrates the occurrence or non occurrence of spikes while the bottom plot shows the corresponding spike magnitudes. From the plots of Figure 6, it can be observed that there is a tendency for spikes to occur in clusters, i.e., there are specific periods within the time series where the probability of spike occurrence is clearly higher than others. This characteristic can be explained by the underlying process driving market prices, i.e., the balance between supply and demand. Periods of high prices, i.e., continuously occurring spikes, are usually the result of a short-term imbalance between supply and demand, e.g., due to fuel shortages or unforeseen shutdowns of large generators.

3. Methodology

3.1. Notation

Indexes a, t, and d are used to denote the area, trading time period, and day, respectively, so that $y_d^{a,t}$ denotes the day-ahead MCP for area a time period t and day d. We define the price spike threshold as $y^{\delta }$ so that a spike occurs in area a, trading period t, and day d if $y_d^{a,t}>y^{\delta }$ and vice versa. The occurrence (presence or absence) of a spike is then given by a binary variable $u_d^{a,t}$ as shown in Equation (2). Similarly, the magnitude of a spike $x_d^{a,t}$ is given by:

$$x_d^{a,t}=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}{u}_d^{a,t}=0\hfill \\ y_d^{a,t}-y^{\delta },\hfill & \mathrm{if}{u}_d^{a,t}=1\hfill \end{array}\right.$$

(3)

3.2. Hawkes Model

The Hawkes model is typically used to model “self-exciting” point processes, i.e., a process in which an arrival of an event increases the rate of future arrivals for some period of time [41,42]. In the case of the electricity price data, we modeled sudden “jumps” (spikes) in prices as excitation signals that increase (or reduce) the price for some period of time after the occurrence.

Fundamentally, we propose the use of the equations proposed by Hawkes to model the price spikes time series data [33]. The Hawkes model defines the occurrence of an event in terms of an intensity function given by:

$${\lambda }_d^{a,t}={\mu }^{a,t}+\sum _{d_i=0}^d{\varphi }^{a,t}(d-d_i)$$

(4)

where ${\mu }^{a,t}\ge 0$ is the base intensity for area a and time slot t while the second term on the right-hand side of (4) is the self-excitation component modeling the influence of past events on the current value of the excitation function and therefore governing the clustering of the point process. As is typical in literature, we used an exponential decay of the form:

$${\varphi }^{a,t}(d-d_i)=u_{d_i}^{a,t}·{\gamma }^{a,t}·e^{-(d-d_i)/{\tau }^{a,t}}$$

(5)

as the excitation function where ${\gamma }^{a,t}$ and ${\tau }^{a,t}$ are model parameters. ${\gamma }^{a,t}$ governs the magnitude of the increase in the intensity function whenever there is a spike, i.e., $u_d^{a,t}=1$, while ${\tau }^{a,t}$ is the time constant for the decay of the effect of the spike thereby giving some information on the influence of the price spike on future event occurrences. The value of the intensity function ${\lambda }_d^{a,t}$ defines the probability of occurrence of a spike in area a at time period t on day d. The price spikes time series can therefore be thought of as stemming from a parameter-varying Poisson process, where $u_d^{a,t}$ is a Bernoulli random variable occurring with the probability ${\lambda }_d^{a,t}\le 1$ (conversely, not occurring with the probability $1-{\lambda }_d^{a,t}$).

Replacing (5) in (4), we can rewrite the Hawkes model as:

$$\begin{array}{cc}\hfill {\lambda }_d^{a,t}& ={\mu }^{a,t}+\sum _{d_i=0}^d{u}_{d_i}^{a,t}·{\gamma }^{a,t}·e^{-(d-d_i)/{\tau }^{a,t}}\hfill \end{array}$$

(6)

$$\begin{array}{cc}& ={\mu }^{a,t}+\sum _{d_i=0}^{d-1}{u}_{d_i}^{a,t}·{\gamma }^{a,t}·e^{-(d-d_i)/{\tau }^{a,t}}+{\gamma }^{a,t}{u}_d^{a,t}\hfill \end{array}$$

(7)

$$\begin{array}{cc}& ={\mu }^{a,t}+e^{-1/{\tau }^{a,t}}\sum _{d_i=0}^{d-1}{u}_{d_i}^{a,t}·{\gamma }^{a,t}·e^{-(d-d_i-1)/{\tau }^{a,t}}+{\gamma }^{a,t}{u}_d^{a,t}\hfill \end{array}$$

(8)

$$\begin{array}{cc}& ={\mu }^{a,t}+{\alpha }^{a,t}\left({\lambda }_{d-1}^{a,t}-{\mu }^{a,t}\right)+{\gamma }^{a,t}{u}_d^{a,t}\hfill \end{array}$$

(9)

$$\begin{array}{cc}& ={\alpha }^{a,t}{\lambda }_{d-1}^{a,t}+{\beta }^{a,t}+{\gamma }^{a,t}{u}_d^{a,t}\hfill \end{array}$$

(10)

where ${\alpha }^{a,t}=e^{-1/{\tau }^{a,t}}$ and ${\beta }^{a,t}=(1-{\alpha }^{a,t}){\mu }^{a,t}$.

The above form of the Hawkes model reveals a structure similar to a typical autoregressive model with ${\beta }^{a,t}$ as the constant term, ${\alpha }^{a,t}$ as the autoregressive coefficient, and ${\gamma }^{a,t}{u}_d^{a,t}$ as the noise term. Rewriting the Hawkes model in this form reveals a more intuitive understanding of the model. In the absence of a spike, the intensity function drops back towards its natural value ${\mu }^{a,t}$ at a speed dictated by the coefficient ${\alpha }^{a,t}$, while when a spike event occurs, the intensity function experiences a jump governed by the value of ${\gamma }^{a,t}$.

3.3. Modified Hawkes Model

In the original Hawkes Equation (6), the parameter ${\gamma }^{a,t}$ that defines the increase in intensity after the occurrence of a spike was taken as a constant value implying that any observed price spike would lead to an increase in the spike’s occurrence probability by the same magnitude. The price spikes time series, however, exhibits a correlation between the magnitude of the spikes and the number of subsequent spikes suggesting a dependence of the underlying intensity function on the magnitude of observed spike events. Therefore, while testing the suitability of the original Hawkes model (6), denoted as Hawkes-I, on fitting the price spikes time series, we also propose implementing two variants of the basic model.

The first of these, denoted as Hawkes-II, is a variable-intensity jump model where the parameter ${\gamma }^{a,t}$ in (6), which defines the increase in the magnitude of the intensity function given a spike, is dependent on the magnitude of the price spike, i.e.,

$${\gamma }_d^{a,t}={\gamma }^{a,t,0}\left(1-e^{-x_d^{a,t}/x_0^{a,t}}\right)$$

(11)

where $x_0^{a,t}$ is the expected value of $x^{a,t}$ given as the average magnitude of price spikes observed up to day d. With this formulation, a large magnitude spike, which typically indicates a significant stress on the supply–demand balance in the market, will result in a relatively larger jump in the magnitude of the intensity function when compared to a spike of lower magnitude.

The second variation, denoted as Hawkes-III, is a variable-effect decay speed model where the parameter ${\tau }^{a,t}$, which defines the rate of decay of the intensity function given a spike, is dependent on the magnitude of the price spike, i.e.,

$${\tau }_d^{a,t}={\tau }^{a,t,0}\left(1-e^{-x_d^{a,t}/x_0^{a,t}}\right)$$

(12)

This formulation suggests that the effect of a large magnitude spike will last longer than that of a relatively smaller spike event.

3.4. Parameter Extraction

We took a Bayesian approach to estimating the Hawkes model parameters. The model parameters were treated as random variables for which we estimated posterior distributions based on the observations up to a given day. The joint posterior distribution of the model parameters for area a and time slot t, given a set of observations up to day d, $p\left({\mathsf{\Theta }}_d^{a,t}\right)$ was given by:

$$p\left({\mathsf{\Theta }}_d^{a,t}\right|X_d^{a,t})=\frac{p\left({\mathsf{\Theta }}_{d-1}^{a,t}\right|X_{d-1}^{a,t})·p\left(x_d^{a,t}\right|{\mathsf{\Theta }}_{d-1}^{a,t})}{\mathsf{\Delta }}$$

(13)

where the prior $p\left({\mathsf{\Theta }}_{d-1}^{a,t}\right|X_{d-1}^{a,t})$ is the posterior distribution on day $d-1$, and $p\left(x_d^{a,t}\right|{\mathsf{\Theta }}_{d-1}^{a,t})$ is the likelihood of the observation $x_d^{a,t}$ given the parameters on day $d-1$. Intuitively, the likelihood function was obtained from the Hawkes model as:

$$p\left(x_d^{a,t}\right|{\mathsf{\Theta }}_{d-1}^{a,t})=\left\{\begin{array}{cc}0,\hfill & x_d^{a,t}<0\hfill \\ 1-{\lambda }_d^{a,t},\hfill & x_d^{a,t}=0\hfill \\ {\lambda }_d^{a,t},\hfill & x_d^{a,t}>0\hfill \end{array}\right.$$

(14)

Starting from a uniform prior distribution on day 0, we iterated through Equations (13) and (14) to obtain the posterior distribution on day d.

3.5. Short-Term Forecasting

Given the magnitude of the intensity function ${\lambda }_d^{a,t}$ on day d, and the model parameters ${\alpha }^{a,t}$, ${\beta }^{a,t}$ and ${\gamma }^{a,t}$, its n-days ahead forecast ${\widehat{\lambda }}_{d+n}^{a,t}$ was obtained by iterating through the equation

$${\widehat{\lambda }}_{d+k}^{a,t}={\alpha }^{a,t}{\widehat{\lambda }}_{d+k-1}^{a,t}+{\beta }^{a,t}+{\gamma }^{a,t}{\widehat{u}}_{d+k}^{a,t}$$

(15)

for $k=1,2,3,\cdots ,n$, where ${\widehat{u}}_{d+k}^{a,t}={\widehat{\lambda }}_{d+k-1}^{a,t}$ and ${\widehat{u}}_{d+1}^{a,t}={\lambda }_d^{a,t}$. Since the spike occurrence forecasting is essentially a binary classification problem, the binary forecast for the occurrence of a spike ${\overline{u}}_{d+n}^{a,t}$ is arrived at by comparing the forecast value ${\widehat{u}}_{d+k}^{a,t}$ to a predefined threshold $\delta$ so that:

$${\overline{u}}_{d+k}^{a,t}=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}{\widehat{u}}_{d+k}^{a,t}\le \delta \hfill \\ 1,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(16)

The decision threshold $\delta$ adjusts the conservativeness of the forecasting model. As $\delta$ tends to zero, the model will forecast more 1’s, which will reduce false negative errors and vice versa.

4. Results

4.1. Data

The JEPX electricity market price data used in this study are publicly available on the exchange’s website [40]. We used the value of 25 JPY/kWh as the price spike threshold, though the results would be similar for any other reasonable threshold values. The modeling and analysis was carried out using data for the 6-year period spanning the start of the fully deregulated market on 1 April 2016 to 31 March 2022—the end of the 2021 Japanese financial year.

Figure 7 shows the spike occurrence probability for two selected study areas (Tokyo and Kansai) over the study period. The spike occurrence probability was calculated simply as the ratio of the number of price spike events in the time series to the total number of events. Figure 7 shows that the spike occurrence probability was generally lower than 3% apart from during the evening peak periods where it rose to a maximum of around 6%. The probability was, however, clearly higher in the Tokyo region (in the 50 Hz eastern grid) than in Kansai (part of the 60 Hz western grid).

Generating individual models for each trading time period for each of the nine areas resulted in 432 different models. Since it was impractical to display the results for all models in this paper, we selected four representative models for detailed discussions. The selected models corresponded to two representative areas—Tokyo for the eastern grid and Kansai for the western grid—and two trading time periods—08:00–08:30 (time code 15) representing the morning peak time period and 18:00–18:30 (time code 37) representing the evening peak time period. These selections gave four area–time-code combinations referenced as Tokyo-15, Tokyo-37, Kansai-15, and Kansai-37 in the following discussions.

4.2. Baseline Model—Persistence Model

We used a persistence model to get baseline results for forecasting accuracy to which the performance of the proposed models were compared. The persistence model used the prices on a given day d to forecast prices over the next N days under the assumption that the present day conditions persisted over the forecasting horizon. For the price spikes time series, the algorithm stated that if a spike occurred in area a, time code t, on day d, the spike was forecasted to persist over the next N days. Mathematically, we wrote this as:

$${\widehat{x}}_{d+i}^{a,t}=x_d^{a,t}\mathrm{where}i=1,2,\cdots ,N$$

(17)

The persistence model estimated the intensity function as the conditional probability of observing a price spike given the last observation. This was calculated as the ratio of the number of spike events that followed the last made observation $u_d^{a,t}$ for the spikes data observed up to day d. While quite simple in formulation, the persistence model could achieve very good performance for such data where correlations between consecutive observations were high and could set relatively high benchmarks for other more complex models under consideration.

4.3. Model Performance: Goodness of Fit

Following the Bayesian approach, we extracted the parameters for the three versions of the Hawkes model. The model parameters were re-estimated daily for the entire dataset with the first two years of data used to generate the first set of parameter values. We used the daily updated model parameters to generate 14-day-ahead forecasts for price spike events as described by (16) with a classification threshold of $\delta =0.5$.

Using the extracted parameters, the intensity function evolution was evaluated for the dataset. Figure 8 shows the evolution of the Hawkes model intensity ${\lambda }_d^{a,t}$ for the four selected area–time-code combinations. These plots show near zero-values during no-spike periods highlighting the rare nature of price spikes during “normal” market conditions. The plots also clearly highlight periods of increased stress in the system with high spike occurrence probabilities. An interesting observation is the prolonged high intensity period that lasts from October 2021 to March 2022.

We showed the goodness of fit of the models on the training data by evaluating the respective values of the log-likelihood functions. Given the values of the intensity function ${\lambda }_d^{a,t}$ and the observations $u_d^{a,t}$, the log-likelihood function for the process that generated the observations was given by:

$$logL=\sum _{d_i=0}^d{u}_{d_i}^{a,t}log{\lambda }_{d_i}^{a,t}+(1-u_{d_i}^{a,t})log(1-{\lambda }_{d_i}^{a,t})$$

(18)

We evaluated the values of $logL$ for the persistence model and the three proposed Hawkes modeling approaches and the results are shown in

Table 2. Values of the log-likelihood $logL$ corresponding to the four investigated models.

log L	Tokyo-15	Kansai-15	Tokyo-37	Kansai-37
Persistence	− 207.7	−204.5	−333.9	−271.3
Hawkes I	−141.9	−147.8	−256.4	−209.9
Hawkes II	−133.0	−135.4	−248.2	−199.6
Hawkes III	−200.9	−198.6	−317.1	−263.4

.

From Table 2, we can clearly deduce that the Hawkes modeling approaches outperformed the persistence model in capturing the underlying characteristic of the spike occurrence intensity. Amongst the Hawkes models, the Hawkes II model consistently outperformed the other versions for all area–time codes. This result suggests that the magnitude of price spikes had a stronger effect on the change in magnitude of the intensity function ${\gamma }^{a,t}$ as captured by the Hawkes II model than on the autocorrelation effect captured by the decay parameter ${\tau }^{a,t}$ in the Hawkes III model. In fact, the Hawkes III model performed worse than the original Hawkes I model and quite similar to the baseline persistence model.

4.4. Model Performance: Spike Event Occurrence Forecasting

While the log-likelihood values of Table 2 give an indication of the closeness of the generated intensity function to the day-ahead price spike occurrence, they do not give an insight on the ability of the models to generate good short-term price spike forecasts. We assessed the forecasting performance of the models based on the ability to forecast the occurrence of spike events for a number of days ahead.

The first metric that we report on the forecasting performance of the studied models is the mean absolute error (MAE) of the intensity function (spike occurrence probability) time series. The MAE is given by:

$$\mathrm{MAE}=\frac{{\sum }_{d=1}^{N_d}{\sum }_{k=1}^{N_k}|{\widehat{u}}_{d+k}^{a,t}-u_{d+k}^{a,t}|}{N_k·N_d}$$

(19)

where ${\widehat{u}}_{d+k}^{a,t}$ is the forecasted value of the intensity function, $u_{d+k}^{a,t}$ is the spike occurrence variable (one for a spike event and zero for no spike event), $N_d$ is the number of days in the dataset, and $N_k$ is the length of the forecasting horizon in days. The MAE values are tabulated in

Table 3. Mean absolute error values of the intensity functions for the four investigated models.

MAE	Tokyo-15	Kansai-15	Tokyo-37	Kansai-37
Persistence	0.0730	0.0738	0.1331	0.1112
Hawkes I	0.0733	0.0771	0.1343	0.1168
Hawkes II	0.0676	0.0702	0.1251	0.1045
Hawkes III	0.0741	0.0806	0.1301	0.1201

for the four candidate models. This metric gives the average deviation of the magnitude of the intensity function from the actual observations. Similar to the results of the log-likelihood values shown in Table 2, the Hawkes II model was found to outperform the other models in generating spike occurrence probability values close to the observations.

The k-day-ahead spike occurrence forecasting results were categorized as true negatives (TN), i.e., ${\overline{u}}_{d+k}^{a,t}=0$ and $u_{d+k}^{a,t}=0$, true positives (TP), i.e., ${\overline{u}}_{d+k}^{a,t}=1$ and $u_{d+k}^{a,t}=1$, false negatives (FN), i.e., ${\overline{u}}_{d+k}^{a,t}=0$ and $u_{d+k}^{a,t}=1$, and false positives (FP), i.e., ${\overline{u}}_{d+k}^{a,t}=1$ and $u_{d+k}^{a,t}=0$. The typical performance index is the accuracy, which measures the number of true forecasts in the dataset given by:

$$\mathrm{ACC}=\frac{TN+TP}{TN+FN+FP+TP}=1-\frac{FN+FP}{TN+FN+FP+TP}$$

(20)

The accuracy index for a time series of binary variables is equivalent to one minus the mean absolute percentage error (MAPE) for a time series of continuous variables. However, we noted that false negatives results would typically have a bigger impact on the bottom lines of market participants than false positives. To capture this characteristic, we adopted the use of a slightly different weighted accuracy (WACC) metric given by

$$\mathrm{WACC}=1-\frac{a\times FN+(2-a)\times FP}{TN+FN+FP+TP}$$

(21)

where $1\le a\le 2$ is the weight placed on the false negative observations. The larger the value of a, the greater the weight placed on the false negative errors and vice versa. In our analysis, we used a value of $a=1.6$ weighting the false negatives four times more than the false positives in (21).

We calculated the WACC for the forecasting performance of the four models in generating 14-day-ahead forecasts and the results are given in

Table 4. Values of weighted accuracy corresponding to the four investigated models.

WACC	Tokyo-15	Kansai-15	Tokyo-37	Kansai-37
Persistence	0.9419	0.9384	0.8929	0.9120
Hawkes I	0.9441	0.9435	0.8923	0.9164
Hawkes II	0.9466	0.9468	0.8980	0.9211
Hawkes III	0.9422	0.9401	0.8947	0.9128

. The results show that the Hawkes II model outperformed the other models in all cases. However, the performance of the Hawkes I and Hawkes III models were very comparable to that of the persistence model. It is important to note the high values of WACC (>0.89) due to the large number of true negatives in the forecasts. This, however, means that even the seemingly slight improvements shown in Table 4 correspond to significant reductions in the number of false negatives in the generated forecasts.

The results shown in Table 4 were calculated for the 14-day forecasting horizon. Figure 9 shows the variation in forecasting performance as the forecasting horizon increases. As expected, the weighted accuracy dropped as the forecasting horizon increased. However, in all four cases, the Hawkes-II model was generally better than the other three models. It is also noticeable that the forecasting performance was almost the same for the one-day-ahead forecasts. The Hawkes-II model was generally better for longer forecasting horizons confirming that it performed better at capturing short-term variations in the intensity function characteristics.

We also assessed the model’s forecasting performance using the Matthews correlation coefficient (MCC), which is typically used to measure the quality of binary classifications [43]. The MCC is similar to the typical Pearson correlation coefficient for continuous variables and is given by:

$$\mathrm{MCC}=\frac{TP\times TN-FP\times FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}$$

(22)

The MCC gives the correlation between observed and predicted variables and takes values between −1 and +1. As a performance metric, it is generally regarded as a balanced measure which can be used even with unbalanced classes as is the case with the price spikes dataset [44].

Table 5. Values of Matthews correlation coefficients corresponding to the four investigated models.

MCC	Tokyo-15	Kansai-15	Tokyo-37	Kansai-37
Persistence	0.5937	0.5845	0.5917	0.6576
Hawkes I	0.6043	0.6065	0.592	0.6676
Hawkes II	0.6317	0.6339	0.6167	0.6915
Hawkes III	0.5995	0.5903	0.6034	0.6569

shows the values of the MCC for the forecasts generated by the test models. As seen from the table, and confirmed by the previous performance metrics, the Hawkes II model generally outperformed the other models in this metric as well.

Similar to Figure 9 showing the weighted accuracy metric against the length of the forecasting horizon, we also plotted the MCC for the 14-day-ahead forecasts in Figure 10. These plots mirrored the results shown in Figure 9 indicating that the Hawkes II model outperformed the other models especially for longer forecasting horizons. The MCC fell from between $0.7$ to $0.8$ for one-day-ahead forecasts to between $0.5$ and $0.6$ for 14-day-ahead forecasts with the averages of between $0.6$ and $0.7$ shown in Table 5.

5. Conclusions

We proposed two variations of the classical Hawkes model for modeling the price spikes time series in the Japanese Electric Power Exchange (JEPX). The first variation modeled the change in the magnitude of the underlying intensity as a function of the magnitude of the price spike while the second variation modeled the change in the decay rate of the underlying intensity as a function of the magnitude of the price spike. From an analysis of the goodness of fit to the training data of the original Hawkes model, the proposed variations compared to a baseline persistence model showed that the variable magnitude variation of the Hawkes model best captured the underlying characteristics of the process generating the price spike events. This was illustrated by achieving lower log-likelihood values compared to the other three models. The modified Hawkes model also performed best in generating short-term (a few days ahead) forecasts of the occurrence of price spike events. The improved performance was demonstrated using three metrics: (1) the MAE of the spike occurrence probability, (2) a modified accuracy index that weighed false negative forecasts more than false positives, and (3) the Mathews correlation coefficient (MCC) that tested the correlation between predictions and observations. The modified Hawkes model especially outperformed the other candidate models as the length of the forecasting horizon increased.

In this paper, we modeled the price spikes events time series—a binary variable indicating the occurrence of a price value exceeding a prespecified threshold. The resultant forecasts gave an indication of the probability of occurrence of a spike without information on the expected magnitude of the spike. As a future research direction, this work could be extended to predict spike magnitudes as well, though this would require some hybridization with more traditional continuous time series modeling approaches. In addition, exogenous inputs such as the forecasts of the availability of large generators and longer-term forecasts of extreme weather conditions—where such data are available—could also be incorporated into the model with a view to improving forecasting performance.