**Evolving Bidding Formats and Pricing Schemes in USA and Europe Day-Ahead Electricity Markets** †

#### **Ignacio Herrero 1, Pablo Rodilla <sup>2</sup> and Carlos Batlle 3,4,\***


Received: 21 June 2020; Accepted: 28 July 2020; Published: 24 September 2020

**Abstract:** This paper compares the evolution of USA and European power markets and evaluates the suitability and future challenges of their designs in the context of the transition to a low-carbon power system. The analysis focuses on bidding formats (the way in which organized electricity markets allow participants to express their operational constraints) and pricing schemes (how agents recover their short-term costs from market prices). The radical evolution of the power mixes worldwide already experienced in the last decade and the larger one to come, with even greater shares of renewable energy and a more prominent role for storage resources, exposes limitations in current market designs. We develop an in-depth and comprehensive review of best practices from both sides of the Atlantic, and learning from them, we draw recommendations to evolve these market design elements.

**Keywords:** wholesale electricity markets; market design; bidding formats; pricing rules; renewable energy sources

#### **1. Introduction**

In the context of liberalized electric power systems, organized short-term electricity markets (as, for instance, the ones run by power exchanges in the European context (See www.europex.org)) not only help participants manage their risks, but mainly serve as a tool to facilitate an efficient matching of supply and demand, ideally contributing to the goal of maximizing market welfare. While electricity is often defined as a commodity (in the sense that one MWh of electricity is indistinguishable from another), experience has shown that for electricity markets to perform these tasks—aligning the economic utility functions of market agents and the physical constraints conditioning supply, it is more than suitable to allow for some complexity to the bidding and clearing procedures.

In markets for most commodities, only the willingness to buy/sell is relevant, but in the case of electricity, a proper consideration of the physical and economical constraints of agents is instrumental to achieve efficient clearing results. Bidding formats allow agents to express in a complex format their willingness to buy or sell electricity, reflecting how their particular constraints lead to the need to respect quantities and intertemporal links. Two very different approaches have been followed in the USA and Europe as regards to how to design these bidding formats, and in both cases, these formats are experiencing limitations to deal with the new paradigm.

Pricing electricity poses several challenges, mostly derived from the presence of non-convexities (such as non-convex costs). As is well known, in a non-convex context, there may be no linear prices (constant prices that remunerates all quantities) that are able to support a competitive market equilibrium. To deal with this issue, again, USA and Europe have opted for different approaches, and in both cases, these different schemes are being challenged by the penetration of new resources.

Bidding formats and pricing rules are key market design elements to allow for an efficient and massive integration of new resources such as renewables and battery storage systems. The objective of this paper is to develop an in-depth analysis on these elements, including the latest and most up-to-date discussions and challenges in USA ISOs (Independent System Operators) and EU Power Exchanges at the time of this writing. The paper is structured as follows:

In Section 2 we describe how USA and European markets followed very different approaches to the design of bidding formats and pricing rules.

Section 3 describes the context that motivates the evolution of power market design. Power system operation is becoming increasingly complex by the introduction of renewable energy resources, and new market players with new requirements are gradually entering into play, such as storage resources and aggregators. Other studies have discussed the impact of these changes in power markets performance, e.g., Hu et al. [1] and IRENA [2] develop comprehensive reviews on overall system needs, and Anuta et al. [3] focuses on the specific case of storage, but this paper focuses on bidding formats in greater detail and encompasses power system resources with more generality.

Section 4 analyzes performance implications of alternative designs and explores potential improvements for current bidding formats, especially in the European context where more limitations have been identified. We also assess other key elements linked to bidding formats, such as the design of clearing and pricing rules. Section 5 provides final conclusions and recommendations.

#### **2. Bidding Formats and Pricing Rules in USA and the EU: Two Di**ff**erent Approaches**

USA- and EU-organized power markets, from their initial implementation, opted for significantly different approaches to design their bidding formats and market clearing rules. The reasons for these differences were diverse, but maybe the most relevant one was the fact that from the very beginning, USA markets run by Independent System Operators (ISOs) were based on a pre-existent integrated structure (the Regional Transmission Operators) who had the responsibility to determine the economic dispatch in a centralized way. Meanwhile, in Europe, the market implementation was focused on prioritizing the economic dispatch (previously, run independently by different utilities) in a single market supported by Power Exchanges (initially, mostly national in scope and mostly non-compulsory). Green [4] and Conejo and Sioshansi [5] develop good descriptions of the fundamental differences of both approaches.

#### *2.1. Markets in the United States: Resource-Specific Bidding Formats*

USA markets use multi-part offers, which explicitly reflect generating units' operational and opportunity costs (such as start-up costs), and their technical constraints (e.g., ramp rates). Multi-part offers are clearly motivated by the market clearing approach adopted by ISOs, which is nothing other than the straightforward application of the Security Constrained Unit Commitment and Economic Dispatch optimization models used before the liberalization of the power industry. Table 1 highlights the typical offer parameters that ISOs make available to thermal units (see, for example, exhibit 4–6 at MISO (Midcontinent Independent System Operator) [6]).


**Table 1.** Typical multi-part offer structure in ISO (Independent System Operator) markets.

In some cases, additional parameters allow multi-stage resources to represent their different operating regimes, and transition costs and constraints between modes (i.e., combined cycle gas turbines, which allow multiple configurations of their gas and steam turbines, and therefore have multiple commitment decisions to take). The bidding parameters highlighted here focus on the energy market, but USA markets also optimize operating reserve provision, and other bidding parameters are provided to this end. So-called flexibility products (which are close in nature to reserve products) are also out of the scope of this paper. Jacob [7] presents a good review of current discussions around flexibility products.

The archetypical multi-part offer is the thermal unit bidding format (predominant type in USA systems), but other bidding formats have also been implemented for different types of resources. For instance, in Section 3.2, we describe recent developments to improve bidding formats available to pumped-storage hydro and other storage units. Not all market agents require complex multi-part offers, and it is possible to submit only price-quantity bids, which could be the case for renewable generators and load serving entities.

In summary, ISOs attempt to represent the power system with the highest detail possible in their clearing algorithm, including the technical characteristics of each generator individually, apart from all the constraints required to ensure reliability. This complex model allows ISOs to optimally schedule resources, while enabling competition in the provision of energy and electricity services.

#### Pricing Approach

These multi-part bidding formats make the clearing problem non-convex, causing well-known challenges in the computation of marginal prices [8–15]. The basic matter of this non-convexity is that the marginal cost of the system may be lower than the average cost of some units. For example, a power plant may be block-loaded, meaning its minimum output constraint is equal to its maximum capacity. These units are usually fast-start gas turbines that only operate economically at full load. A block-loaded unit may be committed by the clearing algorithm but, because it cannot supply the next marginal increment of load, it cannot set marginal prices. In this case, a lower-cost unit could set the marginal cost of the system, making the market price lower than the average cost of the block-loaded unit. For this reason, marginal prices are complemented with uplift payments, which are separate payments that compensate generators for the costs incurred above the revenue earned through market prices. Uplift payments are also referred to as side-payments or make-whole payments.

Uplifts are unavoidable in an optimal dispatch-based market, required to support the welfare, maximizing commitment and dispatch (i.e., to provide a remuneration aligned with dispatch orders). The underlying problem with uplift payments is that they create a discriminatory pricing regime, where not all agents benefit from the same prices, potentially creating misaligned incentives. This means price signals do not fully reflect operational costs, which can also have an effect in long-term investment decisions [16].

Pricing in USA markets, especially in recent years, has deviated from pure marginal costs in an attempt to reduce the weight of uplift, and to internalize, as much as possible, all operational costs into market prices. A notable example is the "hybrid pricing" approach first implemented in 2001 by NYISO (New York Independent System Operator) [17], and that is continuously updated and under review [18–20]. The NYISO pricing approach allows block-loaded units (as the one in the previous example) to artificially become marginal in an ex-post run of the dispatch problem, where the inflexible bid is treated as flexible (as if it could be dispatched at any level between zero and its maximum power output). This way, block-loaded units can set prices, although NYISO only applies this method for a subset of fast-start units.

The more general term used for this practice is Integer Relaxation (IR), which involves relaxing binary constraints in an ex-post pricing run of the commitment and dispatch problem (see [12] or [21] for more detailed discussions). However, the exact method is more nuanced and varies from one ISO to another. Indeed, most ISOs apply some type of IR, but they differ in which units can set prices, and whether they consider start-up and no-load cost in the pricing problem. In some cases (for instance, in the original NYISO hybrid pricing), only the minimum output constraint is relaxed in the pricing run, so only variable costs can impact prices; this practice is frequently called "EcoMin relaxation".

In addition, "fast-start pricing" is also a common term in practice because the relaxation often involves only fast-start units. Allowing fast-start units to set marginal prices can have positive effects, such as sending efficient signals to price-responsive load [22], or incentives to fast-start units to improve their performance or bid their true cost [23]. Fast-start pricing has been a hot topic in USA markets in the last years, with some relevant ISOs not fully allowing prices to reflect the short-term true costs. This led the Federal Energy Regulatory Commission (FERC) on 2019 to, for example, require PJM Interconnection and New York Independent System Operator (NYISO) to "implement tariff changes to ensure their pricing practices for fast-start resources were just and reasonable" [20]. The measures included, among others, using the same time granularity in the ex-post pricing run of the model and in the previous commitment and dispatch problem, or further relaxing the capacity of the fast-start units that are flexible in the price computation.

Fast-start block-loaded resources are certainly a very relevant part of the uplift problem, but this is not the only non-convexity causing price distortions. Start-up and no-load costs of thermal units can potentially cause uplift, both in the real-time and day-ahead markets. A more inclusive approach is applied in MISO (based on a simplified version of Convex-Hull pricing, see [24]); called approximated ELMP (extended locational marginal pricing). This approach is essentially an IR, but it is more comprehensive than NYISO's hybrid pricing. MISO includes start-up and no-load costs in pricing and applies ELMP to all fast-start resources (not only to block-loaded ones). Indeed, MISO broadened the definition of fast-start resources to allow more peaking units to set prices [25]. MISO continues to search for improvements to its pricing approach; during 2019, MISO studied the practical application of new formulations of the convex envelope for ELMP [26], with the objective of changing the methodology when the cost-benefit analysis was clear.

#### *2.2. Markets in Europe: Abstract Bidding Formats*

European power exchanges follow a completely different approach; their main goal is to provide a platform for market agents to trade electricity, simplifying as much as possible the consideration of physical constraints, under the presumed objective to facilitate trading and maximize market clearing replicability and transparency. System operation is decoupled from the market, and left to transmission system operators, which eventually enforce reliability constraints. This vision shifts part of the responsibility in optimizing the operation of generation resources to market agents and expects them to express their willingness to buy or sell power in a simpler way. For instance, most European power exchanges allow portfolio bidding, i.e., generation companies that own several generation units in the same pricing area can submit combined offers, and then internally decide the operation of each unit to reach the required production.

The basic bidding format in Europe is the price-quantity pair; however, a set of more complex bidding formats (or order types or market products, in the European terminology) is also available, as shown in Table 2.



Hourly step and linear piecewise orders resemble the variable cost component in USA multi-part offers, but in this case, all operational costs must be internalized in the offered price (no additional components such as start-up cost are explicitly considered).

Complex conditions can be added to hourly orders to reflect more sophisticated constraints [27]. The minimum income condition available in the Iberian market can constrain the hourly orders of a unit, so they are only accepted if the income of the resource throughout the day reaches a fixed amount (representing, for example, the start-up cost) plus a variable cost component. The minimum income condition, combined with the load gradient condition, represents some, but not all, of the features of multi-part offers. However, the fixed and variable cost components are not considered for the maximization of market welfare, only to reject some hourly orders when the minimum income condition is not met.

Alternatively, block orders are bids that apply to multiple consecutive periods simultaneously, instead of a single hourly period, and are accepted or rejected based on the average price for those periods [28]. Resorting back to the example of the thermal unit, this could allow offering to start a power plant for a given set of hours, internalizing the start-up cost in the average price. Block orders can be combined using exclusive or linking conditions to represent more complex possibilities.

All order types in Table 2 are implemented in the single clearing algorithm EUPHEMIA (acronym of Pan-European Hybrid Electricity Market Integration Algorithm) [28]. However, the orders available in the power exchange designated in each country (the Nominated Electricity Market Operator, NEMO) differ. The integration of power exchanges through the Price Coupling of Regions (PCR) initiative has achieved some standardization of market products, but for the moment, NEMOs have not fundamentally modified the orders available in their territory. For instance, complex conditions were, and still are, only available in the OMIE exchange (for Spain and Portugal), while Nord Pool (Nordic-Baltic region) and EPEX SPOT (central Europe) allow the use of block orders.

Bidding formats are now regularly reviewed; the first proposal being submitted jointly by all NEMOs dates back from November 2017 [29]. This proposal did not include significant changes besides updating some definitions. For instance, hourly orders were defined as Market Time Unit (MTU) orders, and any references to hourly periods were modified accordingly; this was to allow changes in the definition of MTU in the future (the goal is to move from hourly to 15-min periods). In addition, it generalized some definitions to allow the use of all orders as both supply and demand. For example, the "maximum payment condition" was introduced as the demand-side version of the minimum income condition. It is worth noting that, although the 15-min change has not been implemented at the time of this writing (it is expected for 2021), it currently focuses most of the efforts in EUPHEMIA developments [30].

The day-ahead products were approved by all Regulatory Authorities and agreed to at the Energy Regulators' Forum on 23 January 2018. Every two years, NEMOs shall consult the products that should be included in the day-ahead market. The last consultation proposal in April 2020 [31] includes a number of amendments to the current list of day-ahead market products, with the most relevant being the inclusion of a new complex condition: the Scalable Complex Orders (SCO). Unlike the classical MIC order that imposes a minimum income condition (MIC) expressed using a fixed cost and a variable cost, the Scalable Complex Order does not use the former variable cost, and instead uses the prices of the hourly suborders as variable cost on top of a fixed cost. As pointed out by EIRGRID et al. [32], the theoretical merit of SCO over MIC is to improve EUPHEMIA performance, but this merit can only materialize if the SCO eventually replaces (not complements) the classical complex orders.

#### 2.2.1. Pricing Approach

European bidding formats, although seemingly simpler than their USA counterpart, also create non-convexities in the clearing problem with similar implications in the determination of market prices. However, the market clearing approach is not a pure welfare-maximization; the essence of European markets is to determine the highest welfare solution possible that also meets these two constrains:


The uniform-pricing principle is often rephrased as a restriction that does not allow the existence of paradoxically accepted bids (PAB). PABs would be offers accepted in the market which are unprofitable at market prices, so it can be likened to units that require uplift payments in the USA context. Since PABs are not allowed in European markets, uplifts are consequently not needed either. The second principle refers, in the European terminology, to paradoxically rejected bids (PRB), which are those rejected bids that would apparently be profitable at given market prices [28]. As stated previously, simple bids cannot be paradoxically rejected, but PRBs are allowed in European clearing rules for complex conditions and block orders.

These two conditions applied simultaneously constrain the welfare-maximization problem, leading (by definition) to a generally sub-optimal market welfare. This is a matter of trade-offs; in the European context, uniform-pricing is, as an objective worth, the potential loss in short-term efficiency. Among the advantages of uniform-pricing is that demand and generation interact in the market in equal terms, and it is not necessary to define rules to allocate uplift that would inevitably send inefficient signals.

#### **3. The Increasing Need for More Complex Bidding Formats**

Both USA and European markets feature different kinds of complex bidding formats, revealing the higher complexity of electricity markets compared to other commodities. Both USA and EU day-ahead markets were originally designed for a predominantly thermal generation mix (with some notable exceptions), and most complex bidding formats were justified by the operational characteristics of thermal generation resources. While this is quite clear in USA multi-part bids (see Table 1), complex European bidding formats tackle the same problem from a different angle. Section 3.1 explains how the needs of thermal resources are addressed (with some limitations) by European bidding formats, and how the penetration of renewable energy sources makes these complex bidding formats even more necessary.

The transition to a low-carbon power system will most likely necessitate from new energy resources (such as batteries), which will bring their own operational constraints, requiring, as well, new bidding formats. This further justifies the need for complex bidding formats, as discussed in Section 3.1.3.

#### *3.1. Operation of Thermal Resources*

The following example describes the bidding requirements of thermal resources, building from the simplest bidding format possible, to progressively introducing more complexity as the limitations of the simpler formats arise.

This section focuses especially on the challenges derived from the variability and uncertainty of renewable energy sources. Each step of this sequence faces a trade-off between the operational efficiency lost in day-ahead dispatch decisions from using too simple bidding formats, and the additional computational burden required to introduce complex bids.

#### 3.1.1. Initial Setting

As a starting point, the simplest design possible is a single-period simple auction, where market agents submit price-quantity pairs to express their available production or desired consumption, and their production cost or demand utility. This design takes the assumption that producers' cost structure consists fundamentally only of variable costs, and/or producers are able to properly predict how their plants will be committed (so they can efficiently internalize their non-convex costs in their bids). This could be a reasonable proxy in power systems dominated by thermal power plants and characterized by a rather flat net thermal demand (i.e., hydrothermal systems) or at least characterized by a highly predictable one. This is the foundation of European power exchanges, and, for instance, the Italian day-ahead market which still uses only simple orders (still, the original designs were quickly complemented with intraday markets to allow market agents to rectify dispatch decisions, see, for example, [31]) [33]. If they ever did, these assumptions do no longer hold in practice in the vast majority of European markets, so there is a severe risk that this approach does not provide the most efficient, or even a feasible dispatch.

Traditionally, another way of facilitating dispatch decisions, despite the simplicity of the power exchange, was to allow portfolio bidding. The lack of complexity can be compensated by managing a large portfolio instead of an individual power plant. Large generation companies with sufficiently diverse portfolios mitigate the impact of an infeasible outcome of the market clearing algorithm, since a large portfolio allows "absorbing" potential inefficiencies. However, in this context, this approach is nothing but a potential market barrier for potential new entrants, and eventually an alternative to exercise some degree of market power. By using simpler bids, generation companies benefit from disclosing the minimum amount of information about their operating cost structures, as limiting the amount of information contained in bids complicates monitoring of market power, since it is very difficult to link bid parameters to actual operating costs [27].

#### 3.1.2. Variability

One of the reasons why this simple model can lead to inefficiencies is that, in the real multi-period problem, it cannot reflect constraints coupling different periods. For example, thermal power plants have ramping constraints that make the production available in one period dependent on the production in the preceding and following periods. One of the effects of the introduction of renewable energy sources is an increase of the cycling regime of thermal units [34]; in summary, ramping constraints are expected to be binding more frequently, and not incorporating this constraint in the day-ahead schedule can significantly degrade the efficiency of the dispatch.

Dispatch efficiency could be improved by incorporating ramping constraints in the optimization model (as in USA markets, or using the load gradient condition), at the expense of some market transparency, but this is not the only way to face this problem. For instance, block orders allow bypassing this problem, if used to offer a predefined production profile (the so-called profiled block orders), as shown in Figure 1.

**Figure 1.** Simple block order representing a ramp-constrained production profile.

#### 3.1.3. Uncertainty

Using a block order requires the producer to take, prior to the market clearing, a decision about what would be the best production profile to offer into the market. The underlying assumption has been that producers can easily forecast market conditions (not only market prices, but also the resulting unit commitment), and offer the most efficient production profiles. In reality, the market outcome is uncertain, and generators need to account for the uncertainty of demand forecasts, competitors' bids, and renewable production schedules [35].

Linked and exclusive block orders can mitigate some of this uncertainty, allowing producers to express a wider range of potential operating profiles for the clearing algorithm to choose. For example, Figure 2 shows how two additional block orders (orange and green) can be linked to the previous order to potentially extend the range of hours where the unit is operating. Linked orders can only be accepted if the previous (parent) order is accepted.

**Figure 2.** Linked block orders representing multiple possible production profiles.

The uncertainty associated to renewable production has greatly increased the need to model the complexities of power system operation. In USA markets, day-ahead bidding formats already represent operational constraints with detail, and renewable energy sources (RES) does not involve either a change in the way agents bid in the market nor an increase in the number of bids. However, in European markets, the use of block orders has increased significantly in recent years. Vázquez et al. [36] analyzed this effect for the Spanish case.

Figure 3 shows the average and maximum number of block orders used in the PCR region (data from European Stakeholder Committee of the Price Coupling of Regions [37,38]. Only annually aggregated data was available for the period 2011–2013, and monthly for 2014–2017). Not only has the total number of block orders almost tripled from 2011 to 2017, the use of the most complex block types has had even greater growth. As discussed in Section 4.1.2, the use of block orders and complex conditions is expected to keep on rising in the following years. This represents a major challenge from the computational point of view.

**Figure 3.** Average and maximum daily number of block orders in PCR (Price Coupling of Regions) region. (Flexible orders reported before March 2016 correspond to a definition phased out by Nord Pool (Flexible Hourly Block Order); no data was available for the new flexible orders).

The use of exclusive orders remarks the fact that, in the uncertain context resulting from renewable production, producers cannot easily plan the operation of their units. Exclusive orders allow expressing multiple possible production profiles of which only one can be accepted by the market, therefore, it makes it easier for market agents to make offers for different scenarios. For example, the orders shown in Figure 2 express three different production profiles, which could also be represented by three exclusive orders. Exclusive orders can sometimes express a wider range of possibilities than linked orders, since exclusive orders do not need a common parent block. In the example in Figure 4, a unit does not know what the best time to sell its production is, so it offers three different blocks and the clearing algorithm will select the best one (maximizing market welfare) only.

**Figure 4.** Exclusive block orders expressing multiple production profiles.

#### *3.2. New Bidding Formats for New Resources*

The development of bidding formats has been clearly influenced by the needs of thermal power plants, not only in the USA where multi-part bids are used, as previously described complex European bidding formats have also been tailored to the needs of thermal units.

The transition to low-carbon power systems will likely reduce reliance on thermal resources as other flexible energy technologies enter into play to compensate for the variability and uncertainty of renewables. It is difficult to predict what resources will play that role in the future, but whether it is in the form of batteries, hydrogen or other storage resources, all are clear candidates.

Pumped-hydro storage has been present in power systems for many decades, and for the same reasons that complicate the operations of thermal power plants, the participation of resources with (limited) storage capabilities now requires more complex bidding formats.

The key challenge for storage arises when bidding formats force these resources to decide, in advance, when (in which time intervals) to bid as a generator and when to bid as demand. The volume of generation and consumption possible from a storage resource are interdependent. Bidding is especially difficult for new electro-chemical storage resources (such as Lithium-Ion batteries) because of their limited storage capacity. Grid-scale batteries, due to their high cost, are usually sized to store energy only for a few hours at nominal capacity, while pumped-hydro resources can have up to weekly or monthly planning cycles. Although both resources have limitations to participate in power markets with current bidding formats, small storage resources have clearly more limitations.

#### 3.2.1. Developments in USA Markets

Resource-specific bidding formats in USA markets clearly provide almost perfectly-adapted bidding parameters for a selection of resources, but on the downside, discriminate potential new resources which cannot enter the market with ease until specific bidding formats are designed for them. For instance, pumped-hydro resources have participated in ISO markets for many years, but smaller storage resources (e.g., batteries, flywheels) have different constraints that cannot be represented with existing bidding formats.

The abovementioned created concerns that unnecessary barriers to storage resources were limiting competition, and the FERC initiated a consultation in November 2015, which culminated in Order 841 in February 2018, entitled "Electric Storage Participation in Markets Operated by Regional Transmission Organizations and Independent System Operators" [39]. The requirements most relevant to the topics discussed in this paper are the following [40]:


As it is usually the case, the FERC order allows a high degree of freedom for ISOs to customize rules to their specific context. Therefore, these requirements will not be homogeneous across ISOs, but the rule provides an interesting judgement on what are the most relevant constraints of batteries. Table 3 summarizes the potential bidding format for storage following ISOs' resource-specific approach.

The first characteristic that makes these bidding parameters different from traditional multi-part bids is that it allows for both charging (consuming) and discharging (generating) regimes, in a single bid. Before, storage resources needed to present separate offers as generators and consumers.

Although many of the bidding parameters are equivalent to the ones used in multi-part bids maximum/minimum operating limits, ramp rates and maximum/minimum run times— a new participation model is necessary because the constraints are applied simultaneously for charging and discharging. Furthermore, new constraints are necessary to represent the limited energy storage. The state of charge represents how much energy is stored in a battery with respect to its maximum capacity. The definition of models to manage the state of charge has been one of the most open design elements, and has led to different approaches [41]. All ISOs have implemented, as an option,

the so-called self-schedule model, where storage operators are responsible for dispatching the output (independently from the market clearing algorithm).


**Table 3.** Multi-part bid for storage.

On the other extreme, storage operators may prefer to entirely leave to the ISO the responsibility of achieving a feasible dispatch. In this model (known as ISO-state of charge-management), storage operators would not submit hourly price-quantity offers, but rather, the technical parameters allowed in the market. This model has been implemented in CAISO (California ISO), NYISO, and PJM (but only for pumped-hydro storage).

Other models are possible between these two approaches. For instance, the so-called "self-state of charge management", where the operator of the storage facility is still responsible to achieve a feasible dispatch, but may present simple bids in the market to optimize its schedule. This model has been implemented in CASIO, NYISO, MISO, SPP (Southwest Power Pool) and PJM. This model is fundamentally aimed at the real-time market, where the operator can monitor the state of charge and update market bids.

The deadline for ISOs to implement Order 841 was December 2019 Most ISO/RTOs have achieved compliance with the order, but two are still on track to meet the requirements. The New York ISO (NYISO) requested an extension to 2020 (accepted), and Midcontinent ISO (MISO) to 2022 (also accepted).

#### 3.2.2. Developments in European Markets

Arguably, the approach implemented in EU power exchanges provides a general set of "abstract" bidding formats that can be used by any type of resource. However, the current design was not conceived for storage resources.

As described for the USA context, the main bidding requirement of storage resources is to represent the physical link between supply offers and demand bids. In this regard, linked block orders provide a limited way to represent this constraint. For example, as shown in Figure 5, a storage resource could submit a purchase block order and a linked sell block order. This way, the sell order will not be accepted if the purchase order has not been accepted as well. In other words, the battery will only be discharged if it has been charged before.

This use of linked orders has two main limitations for storage resources. First, market agents must decide, in advance, two potential periods to buy and sell power, so the use of storage is not fully optimized. Potentially, this limitation could be addressed by submitting multiple pairs of linked buy-sell orders, including an exclusive constraint so only one of the pairs is accepted. However, linked and exclusive orders cannot be combined.

Second, the linked order guarantees a feasible schedule (since the battery will not be discharged if it has not been charged before), but because the link can only go in one direction, the parent purchase order could be accepted without the sell order. This would produce a feasible but clearly suboptimal schedule, leaving the battery charged without a commitment to sell its energy. This creates a risk to incur losses if using this bidding format. To address the latter limitation, EPEX Spot introduced a new type of bidding format called "loop order" [42]. Loop orders allow submitting two (and only two) blocks which will be executed or rejected together by the clearing algorithm. The introduction of this new order type highlights that bidding formats need to be continuously updated as the needs of market agents evolve; and shows a shift from the "abstract" bidding format approach to much more resource-specific products.

**Figure 5.** Use of linked block orders by storage resources.

It is worth noting the Nordic power exchange features flexible orders, which is an extremely useful bidding format for limited-energy resources, although it is aimed mainly at storage resources with a longer than one-day planning cycles (such as hydro storage). Flexible orders allow participants to express the maximum volume of energy they are willing to sell, and the clearing algorithm will select its optimal allocation. However, flexible orders share some of the abovementioned limitations for small storage resources. The problem of coordinating the sale and purchase of power within a single day remains because it is not possible to combine flexible and linked or exclusive orders.

The current discussion on the products' definition (the aforementioned consultation process launched in April 2020) includes a number of minor amendments to the products, none of them affecting the capability to properly bid storage resources in the day-ahead market.

#### **4. Challenges Ahead**

As described in the previous section, the operational needs of thermal units in a context of increased renewable penetration, and the potential needs of new energy resources, call for rethinking bidding formats. However, reforming such a fundamental market design element opens some additional debates. Section 4.1 discusses some of the computational challenges that could arise from the introduction of complex bidding formats, and Section 4.2 elaborates on the implications in Europe of introducing complex bidding formats similar to those in the USA.

#### *4.1. Computational Tractability*

In both USA and European markets, the computational complexity of the clearing problem is an instrumental factor that conditions what bidding formats can be implemented in practice. USA markets use market welfare-maximizing optimization models to clear markets—which despite their large size are reasonably tractable problems—followed by separate pricing models. European markets, despite including less detail in modeling the physical constraints of the system, combine clearing and pricing in a single more computationally complex model.

#### 4.1.1. Challenges in USA Markets

As previously discussed, USA markets use detailed multi-part bids, which capture most of the complexity of thermal generating units. This model is well prepared to face the introduction of larger shares of renewable production. ISOs have progressively increased the modeling detail in their markets [43], as made possible by optimization software improvements and developments in computing technology. This does not mean the USA model is not constrained by its computational tractability, but for the moment, computational improvements have continued to allow for incremental modeling enhancements. For instance, some ISOs have already implemented new bidding formats for storage, similar to the one previously described, see e.g., [44,45]. However, computational problems could arise, not because of the complexity of these bidding formats, but due to the larger number of participating resources. New storage resources could be 1 MW or less in size, which is orders of magnitude smaller than traditional resources, meaning the number of market participants could be hundreds of times the current amount. The size of the resulting commitment and dispatch problem could become intractable, and indeed, FERC Order 841 [39] included provisions to allow increasing minimum bid size requirements:

*We are also not concerned about the potential availability of software solutions as multiple RTOs*/*ISOs already provide a minimum size requirement of 100 kW for all resources and have not expressed similar concerns regarding the minimum size requirement. While establishing a minimum size requirement of 100 kW for the participation model for electric storage resources will result in some smaller resources entering the markets in the near term, we do not expect an immediate influx of these smaller resources or any resulting inability to model and dispatch them. However, we recognize this finding is based on the fact that there are currently fewer 100 kW resources than there may be in the future. Therefore, in the future, we will consider requests to increase the minimum size requirement to the extent an RTO*/*ISO can show that it is experiencing di*ffi*culty calculating e*ffi*cient market results and there is not a viable software solution for improving such calculations.*

This computational problem in ISO markets results from the combination of two factors: complex bidding formats and the number of resources. Therefore, the scalability issue can be confronted from both sides. Increasing the minimum size requirement is a way to reduce the number of resources, but this also limits competition, so it is only acceptable as a short-term solution. This measure should be accompanied by the development of rules for the participation of aggregations of resources, which opens all sorts of new questions. For instance, defining bidding parameters for aggregators cannot follow the usual resource-specific approach in ISO markets, since this participation model calls instead for general bidding parameters.

An alternative approach would be to simplify existing bidding formats. Just as creating participation models for aggregations rather than individual resources, this approach would reduce the ability of ISO markets to model the physical system accurately. Taking any of these solutions would significantly change the current modus operandi in ISO markets, but there are several reasons why ISO markets will not need to simplify its approach all the way to European-like bidding formats. As already discussed, the welfare-maximizing clearing approach allows for much more complex bidding formats than the uniform-pricing clearing approach, now and in the future. Furthermore, ISOs have not shown interest in facing one of the greatest challenges of European markets (see next section), which is to integrate several states/countries in a continent-scale market. For now, each ISO market has a well-defined footprint, and although some markets are expanding their geographical scope (for example, California ISO's real-time market has been opened to neighboring balancing authority areas through the Energy Imbalance Market (See www.westerneim.com)), no plausible plans exist to further integrate all North American ISOs. Such an objective in the future, however, would most likely require taking modeling simplifications.

#### 4.1.2. Challenges in European Markets

As discussed in Section 2.2.1, the uniform-pricing rule conditions the clearing problem in European markets. This approach combines clearing and pricing in a single, more computationally complex problem. Computational complexity becomes especially relevant when considering the ultimate goal of European markets is to integrate all European member states in a single clearing algorithm. Computational problems have already surfaced during the first years of operation of the PCR, and Eastern European markets are to join the common platform in the upcoming years. Probably, the most

relevant concern nowadays in European markets is the existence of PRBs. As previously described, PRBs are unavoidable under uniform price-based clearing; however, in certain cases, bids may be incorrectly rejected due to the computational complexity of the algorithm. As pointed out by the Market Parties Platform (MPP) in the European Stakeholder Committee of the Price Coupling of Regions [46]:

*There may exist false PRBs: rejected in-the-money blocks that could have been accepted and result in a better (higher welfare) solution. MPP asks for more transparency on optimality, to prove the absence of false PRBs.*

The reason behind this matter is that, mathematically, the clearing problem is a non-linear and non-convex problem, for which it is difficult to prove the optimality of a solution, or to take a quantitative measure of the quality of a solution. This may hinder the confidence of market participants, together with a lack of clarity in the public documentation of the clearing algorithm [29]. The joint response of ACER (Agency for the Cooperation of Energy Regulators) and CEER (Council of European Energy Regulators) [47] to the European Commission's Consultation on a new Energy Market Design claims that: "We would particularly like to see clearer rules and greater transparency around the market coupling algorithm (EUPHEMIA)".

Computational complexity may limit the scalability of European markets, not only to integrate more Member States in the PCR, but also to cope with the increasing trend in the number of block bids submitted to the market.

Performance and scalability are two pillars that need to be reinforced in the algorithm, and as a consequence, ACER Decision 08/2018 on the Algorithm Methodology on 26th July 2018 established that NEMOs have to report regularly on the following aspects regarding day-ahead market coupling: Operations (incidents and corrective measures), Performance Monitoring (performance indicators, including paradoxically rejected blocks and social welfare), Scalability and R&D.

As regards to the scalability concern and the growing use of block order and complex conditions, the first report, published in November 2019 [48], confirmed the expectation is for the use of both complex and block orders to keep growing in the coming years. Figure 6, from said report, shows the expected usage of complex and block orders in 2021 (expressed as percentage of 2018 usage), with the use of complex orders and linked block orders expected to grow to 170% of its 2018 value. Those estimates did not take into account the potential impact of the forthcoming 15/30-min products, for which proper data sets and specifications were still missing.

**Figure 6.** Usage of complex and block orders in 2021 (expressed as percentage of 2018 usage) [48].

EU power exchanges have traditionally limited the amount of block orders submitted by portfolio (i.e., by market agent and market area), as shown in Table 4. A potential solution would be to further reduce block order limits, but this is clearly not a desirable outcome.


powerexchanges[49,50].

Orders/Portfolio

5

 3

Max.

#### *Energies* **2020** , *13*, 5020

Furthermore, current rules require the clearing algorithm to obtain a solution in less than ten minutes [51]—A much more demanding timeline than USA markets, so the quality of the solution could also be improved by allowing additional time for the clearing process. However, European markets do not incorporate many physical constraints, making additional corrections by Transmission System Operators necessary, so it may not be possible to extend this timeline by a large margin.

Both reducing the number of block orders, and extending the time available to compute the solution, are temporary fixes. In the long term, it is necessary to focus on the root causes. The number of block orders submitted has greatly increased and will keep increasing because no single bidding format properly addresses the needs of market agents, therefore, agents combine orders in an effort to hedge against all the possible market outcomes.

A more permanent alternative would be to create resource-specific bidding formats that would only require one (multi-part) bid per resource. However, since such resource-specific bidding formats would likely be more complex, they should be carefully designed to ensure they indeed reduce the number of orders and overall problem complexity. A currently discussed alternative in this line is the introduction of thermal orders, which is nothing more than multi-part bids like the ones used in ISO markets.

The European Stakeholder Committee of the Price Coupling of Regions [38] suggests that thermal orders can be preferable (from a computational efficiency point of view) if an agent can submit a single multi-part bid replacing multiple block orders. However, in the same stakeholder committee, the EUPHEMIA software provider (N-Side) has also pointed out that including such a bid would need to consider a significant change in the market design, and pricing and clearing rules, which are discussed in the next section.

#### *4.2. Clearing and Pricing Rules*

The same fundamental problem—pricing in non-convex markets—has surfaced in different ways in USA and European markets because of differences in clearing rules. The main challenge in USA markets is to reduce uplift payments, which may be a clearer symptom of inefficient pricing, but similar issues should be expected in the European context. The uniform price-based clearing rule in EUPHEMIA relies on marginal pricing, even if the dispatch solution is not fully welfare-maximizing. This means that, in the EU context, inflexible bids cannot set market prices (similarly to block-loaded units in USA markets). As described by Eirgrid et al. [51]:

*The e*ff*ect of defining an order as a block is that the order cannot then be a full price maker. Rather, block orders may impose a bound on the range of prices possible while the price being set would still need to come from the simple order or complex order curves. This is because the decision to execute the order is an integer decision (i.e., the order is executed or not executed) and the decision on whether to accept a block occurs before the price determination sub-problem. The bound created by the last accepted block order would function to a*ff*ect the price (by limiting possible values) but could not directly set this price.*

*This was discussed with the PCR ALWG representative, APX, who confirmed that without the blocks setting the price, the price could only be set by other price makers, i.e., simple orders or complex orders, or the price indeterminacy rules of EUPHEMIA.*

Another similarity with USA markets is in the way some units may represent their start-up costs using the minimum income condition. This constraint guarantees that the offers of a unit will only be accepted if the price is high enough to compensate a given fixed cost (representing the start-up cost). Although this fixed cost influences the clearing of the market—triggering the rejection of the offer if the price is not high enough—it does not directly set the market price, which is always set by a simple (marginal) bid.

In a scenario where most bids are inflexible, or where they use any non-convex order type, EUPHEMIA can lead to market prices that do not accurately reflect system costs. Note that, even if

uniform prices necessarily include all operational costs, there is a nuanced difference between market prices being high enough to cover costs, and market prices being reflective of system costs. This is, in essence, the same problem described for USA markets, where inflexible units could not set the price. Fortunately, this also means that European markets could benefit from the solutions developed for the USA context. However, the only way to allow alternative pricing rules is to modify clearing rules as well.

Clearing rules in European markets are based on uniform pricing, but this is not imposed by bidding formats. Even though market orders always express their constraints with respect to market prices (i.e., if price is below X, reject order), the uniform-price clearing approach is mostly the result of a market that has evolved from a simple auction. The basic information contained in market orders is the necessary remuneration, but said remuneration could include uplift payments if such a policy choice was made. Obviously, this requires a significant change, but current bidding formats do not prevent using a market welfare-maximization clearing approach and an alternative pricing rule.

Indeed, a welfare-maximization clearing approach would greatly simplify the clearing algorithm, helping European markets cope with the current increase in the use of block orders. Additionally, it would enable more complex bidding formats, such as thermal orders; and more complex combinations of block orders, as made necessary by storage resources.

This discussion had a greater momentum back in 2016 [52] and today is not high up on the agenda of NEMOs. However, the possibility has not been completely discarded, and it is still included among the different R&D activities to be explored [53].

#### **5. Conclusions**

The penetration of renewable energy resources has significantly altered power systems. In light of these changes, wholesale electricity markets, and, in particular, day-ahead markets, in their role to facilitate planning and operating decisions, require increasingly complex bidding formats. While USA markets already provide detailed multi-part bids to reflect the most relevant constraints of thermal generators, European markets provide a limited choice of block orders and complex conditions. These orders may be falling short to facilitate an efficient participation of all resources into electricity markets, as evidenced by the ever more complex combinations of orders submitted by market agents to achieve an adequate representation of their constraints.

The energy transition will also bring about the introduction of new energy resources, for example, batteries and other types of storage, making it necessary to address their needs and remove barriers for effective competition. The definition of participation models for storage is underway in USA markets, but European markets lack specific bidding formats for these types of resources. Although European markets use abstract bidding formats that should not discriminate any resource (meaning, they are equally limited for all types of resources), storage resources face significant barriers.

Since most difficulties have been identified in the European context, this is where the following conclusions focus. Regarding the limits of current bidding formats to represent both thermal and storage resources, a potential solution is to increase the range and complexity of the orders available. However, European markets are already facing computational difficulties, and this approach would most likely fall into scalability issues.

Therefore, the most sustainable approach in the long term would need to both reduce the computational complexity of the clearing problem, and allow more complex bidding formats. This may seem an impossible puzzle, but there are ways in which it can be achieved. First, resource-specific bidding formats, similar to USA's multi-part bids, can, in some cases, reduce the computational burden if one multi-part bid substitutes a complex combination of block orders. At the same time, resource-specific bidding formats would remove barriers for small market players (current portfolio bidding is advantageous for large players), and facilitate market monitoring.

However, the primary cause for the limitations of European bidding formats is the clearing approach. European markets are based on uniform prices; clearing the market under uniform-pricing constraints complicates the computation of market programs, so the range of bidding formats available is limited to keep the computational burden under control. Alternatively, USA markets use computationally simpler clearing algorithms based on the maximization of market welfare (without pricing constraints). As reviewed in previous chapters, this approach requires an ex-post price computation, with its own challenges, but it would enable the use of increasingly necessary complex bidding formats in the European context.

In conclusion, resource-specific bidding formats are most advantageous, especially when combined with welfare-maximization clearing rules. However, their design has to be regularly reviewed to ensure no resources are discriminated. This is especially challenging when considering future potential energy resources, of which their technical characteristics are yet unknown. Nonetheless, this cannot be strictly considered a disadvantage over European (resource-independent) bidding formats, since these are equally influenced by current resource needs, and also become outdated. For instance, European-bidding formats present several limitations to represent the constraints of storage resources, so their resource-independence cannot be unequivocally considered a positive feature, unless full generality is achieved.

**Author Contributions:** I.H. conceived and designed the analysis, collected the data, performed the analysis and wrote the paper. P.R. conceived and designed the analysis, collected the data, performed the analysis and wrote the paper. C.B. conceived and designed the analysis, collected the data, performed the analysis and wrote the paper. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Short-Term Electricity Price Forecasting with Recurrent Regimes and Structural Breaks**

#### **Rodrigo A. de Marcos 1,\*, Derek W. Bunn 2, Antonio Bello <sup>1</sup> and Javier Reneses <sup>1</sup>**


Received: 25 September 2020; Accepted: 15 October 2020; Published: 19 October 2020

**Abstract:** This paper develops a new approach to short-term electricity forecasting by focusing upon the dynamic specification of an appropriate calibration dataset prior to model specification. It challenges the conventional forecasting principles which argue that adaptive methods should place most emphasis upon recent data and that regime-switching should likewise model transitions from the latest regime. The approach in this paper recognises that the most relevant dataset in the episodic, recurrent nature of electricity dynamics may not be the most recent. This methodology provides a dynamic calibration dataset approach that is based on cluster analysis applied to fundamental market regime indicators, as well as structural time series breakpoint analyses. Forecasting is based upon applying a hybrid fundamental optimisation model with a neural network to the appropriate calibration data. The results outperform other benchmark models in backtesting on data from the Iberian electricity market of 2017, which presents a considerable number of market structural breaks and evolving market price drivers.

**Keywords:** day-ahead electricity markets; electricity price forecasting; fundamental-econometric models; market structural breaks

#### **1. Introduction**

Price forecasting in electricity markets is facing frequent, and perhaps increasing, structural changes in the market. Apart from new entrants and corporate restructuring affecting market conduct, the technology mix is going through a transition to intermittent renewables and end-user engagement is becoming substantial. In addition, policy interventions are increasing as governments seek to achieve a balance of decarbonisation, security and affordability. All of this creates a modelling challenge for price forecasting. Time series estimation, therefore, has to take account of structural breaks and evolving parameters as market circumstances change. A simple response is to work with short time series to reflect only recent conditions that may be representative of the intended forecast horizon, but that limits the complexity of model estimation. In contrast, econometric methods often seek to include estimated structural break terms. These, however, tend to be limited to a few distinct interventions and do not capture more complex evolutions. Hybrid methods, alternatively, that link time series analyses to underlying market simulation models can be more effective [1], but even with a hybrid method, the choice of an appropriate time series calibration length still remains.

Surprisingly, despite its crucial role, research on how to select the appropriate data window for model estimation is an under-researched topic in forecasting. Whilst we have seen time series methods increase in complexity to capture the distinctive features of power price formation, going from ARIMA and its variants [2–4], neural network and other AI approaches [5–10] to wavelets [11–13] and various combinations, these procedures all rely on the presumption that the time series model, as estimated, can be projected forward, which may not be so appropriate in the more evolving power systems of today. In one of the few research papers to look at this aspect, the sensitivity of forecast errors to the estimation window has been analysed in [14] and based upon this, the research in [15] presented a pragmatic averaging of forecasts of individual ARX models estimated upon different data calibration windows. However, only heuristic suggestions were made for the choice of windows for calibration.

To complicate the specification further, apart from permanent abrupt and gradual structural changes affecting the window of relevant history, power price formation is known to manifest recurrent regime changes and exhibit multi-seasonal behaviour [16], according to the interactions of periods of scarcity, input prices, weather conditions and behavioural dynamics. Thus we have seen Markov and factor-based regime-switching methods outperform single regime models in several comparative studies [17,18]. The implication of this is that there are recurrent episodes in the time series when one specification is more relevant than another. So, if we are seeking to find the most relevant window of data for model calibration, it may not be the most recent. For instance, if the power system is expected to experience a sharp and sudden increase in wind generation based on weather forecasts, it may prove advantageous to disregard for predictive estimation any periods in the past that do not present significant wind outputs. Our research, therefore, seeks to make a contribution by developing a methodology to select the appropriate calibration window for a hybrid fundamental/timeseries forecasting approach based upon considerations of structural breaks and recurrent regimes. It is evidently important to have an integrated method to select the calibration window both with respect to considerations of recurrent regimes as well as respecting structural changes, and we are not aware of this joint specification being considered in previous research. This is therefore the focus and main contribution of this research.

Various aspects of calibration window selection have appeared in previous research but without the full specification being sought in this paper. For example, in neural networks, [11] uses a training set involving the seven days prior to the forecasting day and adds three extra days based on the similarity with respect to the day immediately prior to the forecasting day in terms of daily price patterns. In contrast, [19] utilises a modified version of the similar days method proposed in [20] in order to select the 12 most similar days in a predefined 4-month calibration period according to exogenous variables available at the moment of the forecast, such as expected demand and temperature. However, these methods are motivated more by considerations of neural network overfitting issues rather than by market regime changes.

In providing a more formal method for calibration window selection, according to robust criteria for identifying both recurrent regimes and structural changes, we undertake this in the context of advocating a hybrid fundamental/econometric approach. We consider the inclusion of a fundamental market simulation model crucial for forecasting with structural changes since it can explicitly represent price formation under new market conditions. For example, the impending decarbonisation of power systems is not a recurring event that econometric approaches can interpret and project in the future. Indeed, that is why hybrid methods have become widely applied in medium-term applications [21–23]. But only a few researchers have considered applying them to the short term [19,24], perhaps because of the high computational requirement of running hourly fundamental models. Nevertheless, this issue can be dealt with by means of simplification methods, such as aggregating similar generation units [24,25]. Furthermore, given that these hybrid models explore most of the drivers of electricity prices, an immense volume of information must be handled by the models. For accuracy, however, this is worthwhile, but only if the time series calibration window is appropriately chosen.

In summary, therefore, this work attempts to provide a novel forecasting method that properly addresses the joint problem of recurrent regimes and market structural breaks in selecting the calibration window, in order to support a state-of-the-art hybrid model. The hybrid model is similar to one of the models proposed in [19] and involves a short-term model that is composed of an hourly cost-production optimisation model whose outputs provide market-related information to a neural network (NN) model. However, there are several distinctive modelling features of this new work that add to its novel research contributions:


The remainder of this manuscript is organised as follows: the methodology is described in Section 2; Section 3 presents the case studies in which the proposed forecasting method, as well as other comparative forecasting methods, have been tested; and Section 4 contains the conclusions, including suggestions for potential extensions.

#### **2. Proposed Methodology**

Essentially, this work's proposed methodology is comprised of the methods displayed in Figure 1, all of which have been tested on a real-size power exchange with complex price dynamics: the Iberian (Spain and Portugal) electricity market. The first phase of the methodology represents its fundamental component, the cost-production optimisation model. The next stage involves several data pre-processing approaches that aim to enhance the final step of the methodology, which is an artificial neural network model. Each element of the proposed methodology is explained below.

**Figure 1.** Overview of the proposed hybrid forecasting methodology.

#### *2.1. Cost-Production Optimisation Model*

In order to consider physical elements, regulatory limits and the operation of the market, a cost-production optimisation model, which is based on the Iberian power exchange, is specified. The required information is obtainable from the transparency platforms of the Spanish System Operator [26] and of the ENTSO-E [27]. This fundamental model seeks to reduce total system costs under perfect competition assumptions by setting the production outputs of the system's power units to optimal values. The mathematical formulation of this optimisation model is similar to the one presented

in [19,25] and estimates the electricity market price as a result of the market-clearing according to competitive fundamentals. These prices are known as system marginal prices, and they are derived from the dual variables of the demand and generation balance constraints. Furthermore, it was observed in [19] that considering thermal units separately is worthwhile in terms of accuracy. Specifically, although a week is solved by minimising system costs simultaneously throughout the 168 h in 7.4 s (up from 3.91 if the thermal units are aggregated), the forecasting error is reduced by approximately 33% when compared to the optimisation model of [25]. The optimisation is solved via relaxed mixed-integer programming (RMIP) in order to consider all units' variable costs and not only those of the committed units.

#### *2.2. Period Selection*

The main contribution of the work, however, is an improvement in model performance by achieving an appropriate calibration data selection procedure. The calibration period selection methodology provides a suitable and novel solution to this issue, allowing the subsequent NN model to handle only the necessary data by focusing upon the relevant circumstances or regimes present in the power system at the moment of the forecast. This methodology is split into three steps.

#### 2.2.1. Structural Breaks

Before applying any filtering method, the initial dataset period needs to be oversized in order to find an appropriate subset. In this case, 13 months prior to the forecasting period are taken (i.e., a 13-month rolling window dataset), which is too large a calibration dataset for NN models if hourly precision is considered. The fact that structural patterns change throughout a 13-month period is not in question. Not only due to several seasonal effects that occur in the system but also abrupt market condition fluctuations or other structural breaks. An example can be seen in Figure 2, which shows the evolution of the Iberian electricity market prices during the autumn of 2016. It can be observed that early autumn is significantly different from late autumn. When it comes to forecasting late autumn prices (e.g., shortly after 6 December), it is evident that one should consider discarding the previous periods with the lowest prices, as they clearly correspond to other market circumstances. It should be noted that the structural breaks depicted in Figure 2 serve as an illustrative example and this work's case study does not involve forecasting prices during late 2016. The different market circumstances are separated by the vertical lines, which correspond to the structural breaks. These structural breaks have been computed by means of the "strucchange" package in R that is based on the work presented in [28].

**Figure 2.** Iberian electricity market prices during autumn 2016.

In theory, structural breaks split a time series into several segments that feature significantly different coefficients and perhaps different model specifications. In this application, we test the baseline model that the electricity price equals a constant. Thus, the "constant" becomes the varying element in the segments that are separated by the structural breakpoints. The purpose of the methodology presented in [28] is the determination of these breakpoints whose corresponding segments provide the least total residual sum of squares of the models associated with each segment.

Evaluating a 13-month hourly dataset is cumbersome if high precision is desired and therefore, the number of candidate breaks should be limited. In order to capture most of the structural breaks in the 13-month price dataset, the breakpoints were computed in two sequential runs. The first run involves a daily arrangement of the 13-month dataset with a minimum breakpoint distance of one week. The second run involves an hourly arrangement of the remaining days as a result of the first run. After computing the breakpoints in a run, the input dataset is divided into periods, which are compared to the most recent period in terms of the average price. In order to discard sufficiently dissimilar periods that belong to other market circumstances, the periods where the price average falls outside the interval μ ± σ, where μ and σ represent the most recent period's price average and standard deviation respectively, are discarded. While larger thresholds than μ ± σ (e.g., μ ± 2σ, μ ± 3σ, etc.) are chosen in other contexts to discard outliers, two periods in time may belong to different market conditions even with a difference of one standard deviation. As a result, this unique manner of performing the methodology of [28] provides an efficient way of detecting structural breaks in a 13-month dataset with hourly precision, as well as discarding significantly different periods as per price behaviours.

Figure 3 depicts the resulting calibration period selection according to the structural break analysis. Whilst the left *y*-axis is related to Iberian electricity market prices from December 2015 up to December 2017, the shaded shape indicates the calibration periods (*x*-axis) selected for a certain forecasting day (right *y*-axis). For example, if the first day of 2017 is selected by drawing an imaginary horizontal line that crosses said day in the right *y*-axis (which, in this case, the line coincides with the upper border of the graph), the shaded area overlaps this imaginary line during 3 months in late 2016 and part of December 2015 according to the *x*-axis, which represents the calibration periods that are selected if the forecasting day is the first day of 2017. Given that early 2017 was characterised by uncommonly high prices, the selected calibration periods were much shorter than those of late 2017. Furthermore, January's peak is generally discarded from calibration datasets when forecasting days later in that year. Moreover, summer 2016 is considered while forecasting summer 2017. Therefore, the result of this algorithm eliminates periods in the past that are expected to be highly dissimilar to the forecasting period.

**Figure 3.** Structural breaks algorithm period selection.

#### 2.2.2. Hourly Clustering

This stage seeks to determine the most relevant factors regarding the market conditions during the forecasting period. The variable with the most predictive content is the estimation of the actual price from the fundamental model, which reflects several aspects of the operations and the dynamics of the market. Although futures prices are often used as predictive variables, they were less useful here than the market-clearing prices, since they do not specify intraday effects. A variable that responds well to sudden market condition disruptions is the expected thermal gap, which represents the difference between the expected demand and the expected renewable generation from wind and solar facilities. Prices are bound to fall if the gap is low. Although the expected market-clearing prices also capture this effect, the expected thermal gap contains a higher level of short-term dynamic information and thus indicates intraday effects with higher definition. The expected temperature is also useful in order to remove periods with significantly different temperature effects.

A K-means clustering method was applied to take these three exogenous variables into account (estimated market-clearing prices, expected thermal gap and expected temperature) and relate the hours in the forecasting period to those of the training period. Given that these exogenous variables are expressed in different units and orders of magnitude, they were standardized before applying the clustering procedure. The K-means clustering application involves the identification of centroids of the values of those three variables throughout the 13-month initial dataset. Consequently, each hour in the dataset belongs to the closest centroid in terms of squared Euclidean distances in the 3D plane formed by the three variables. Depending on the predefined number of clusters, the centroids are placed so as to minimise the total quantisation error or the sum of squared Euclidean distances. Thus, a greater number of clusters lead to lower quantisation errors and higher complexity levels. In order to appropriately set the number of clusters, the K-means algorithm is computed for several numbers of clusters and, by means of a Pareto optimal frontier procedure [29], a suitable compromise between complexity level and total quantisation error is obtained. Finally, the clusters that include the hours of the forecasting period are deemed relevant and thus the hours of the input dataset that do not belong to said clusters are discarded.

The combination of these period selection algorithms is intended to discard the information pertaining to dissimilar market regimes, according to recent price behaviours and forecasted market regime indicators. Therefore, the hours that were not discarded by the structural breaks method were combined with those included by the K-means procedure, as displayed in Figure 4. The difference in Figure 4's shape with respect to that of Figure 3 is related to the hours that are only selected by the K-means method (i.e., there are hours that were chosen by both techniques). This new shaded shape is somewhat hollow given that the clustering has been performed hourly. This provides useful information as to what intraday patterns in the past are the most similar to that of the forecasting period.

The resulting calibration dataset that is shown in Figure 4 contains two sets of information: the recent dynamics such as agent strategic behaviours provided by the structural breaks method and the patterns that are driven by market fundamentals yielded by the hourly clustering technique. All in all, this combined dataset discards the information pertaining to dissimilar market regimes according to recent price behaviours and forecasted market regime indicators in an automated fashion.

**Figure 4.** Addition of an hourly clustering method to the period selection methodology.

#### 2.2.3. Neural Network Validation Set

Considering the length of the filtered dataset, a validation set is obtained following the similar days method performed in [19], which selects the top 20% of days in the most recent segment (i.e., between the most recent structural break and immediately prior to the forecasting period) as per their similarity with respect to the forecasting period in terms of daily patterns regarding exogenous variables such as expected demand.

#### *2.3. Artificial Neural Network Model*

As displayed in Figure 1, four outputs of the fundamental model are combined with common predictors to form the set of input variables for the NN model. This set consists of the following factors:


This set of variables has been obtained and validated by means of a variable selection procedure based on mutual information and partial mutual information in order to analyse their dependency with respect to electricity prices and their redundancy with respect to the other explanatory variables when used to predict electricity prices. However, it is worth noting that this work's contributions are centred on the period selection methods.

Once the proposed period filtering methods have been carried out, the remaining data are used as training inputs to a NN forecasting method. The literature suggests that the most suitable NN configuration is a single hidden and output layer architecture, as stated in [30]. Another well-established choice in the literature is the Levenberg-Marquardt training algorithm. The hyperbolic tangent sigmoid activation function was utilised for the hidden layer's neurons and a pure linear activation function has been resorted to for the output layer. However, due to the lack of consensus in the literature with regards to the number of neurons to be set in the hidden layer, several variations were tested (a range from 10 to 60 with a step of 5 neurons). The validation set mean-square error MSE of the neural networks was used in order to choose the optimal number of the hidden layer's neurons. Moreover, in order to consider the random initialisation of the weights of the NN training algorithm, a high number of replications of the NN forecasting procedure were carried out. This is also done in order to improve the likelihood of the NN training algorithm going from local to global minima.

#### *2.4. Model Performance Metrics and Evaluation Criteria*

Consistent with most forecasting research, the models are evaluated using three error metrics: mean absolute percentage error (MAPE), mean absolute error (MAE) and root-mean-square error (RMSE). Note that electricity prices can sometimes approach zero and in such cases, MAPE may approach infinite values. Nevertheless, the case study utilised in this work does not include any actual zero-price occurrences. In order to assess performance comparisons in a statistically significant manner, the Diebold-Mariano (DM) test has been carried out with a 5% significance level [31].

#### **3. Case Studies, Results and Discussion**

The case study for this work is the Iberian electricity system throughout the entire year of 2017. Early 2017 was characterised by very cold weather, low renewable energy generation, high natural gas prices and external disruptions originated by the decommissioning of nuclear power plants in France. This caused the price surge that is seen in Figure 3, which represents the 2017 maximum price of 101.99 €/MWh (up from 2016's peak of 75 €/MWh). Furthermore, a steady increase in the European CO2 emission allowance prices began in the late summer of 2017. More specifically, these prices rose by approximately 25% throughout the year of 2017. Therefore, this case study poses a highly challenging task with disruptions and evolving changes and is, therefore, a suitable test of the methodology proposed in this paper.

First of all and as per Figure 1, the cost production model is run so as to obtain the necessary information to carry out the remaining stages of the proposed methodology. This provides the fundamental model output variables: market clearing prices as well as CCGT, hydro and coal unit generation output levels. Furthermore, given that the aim of this work is to provide forecasts for the entire year of 2017 and that the NN model's initial training dataset is of 13 months, all input variables must be made available from December 2015 up to December 2017. Once these 13 months are filtered according to the methodologies presented in the previous section, the NN model is run to provide rolling forecasts for every day of the year in 2017. Therefore, the actual forecasting horizon is of one day.

In order to specifically assess the ingredients of this methodology, the proposed hybrid forecasting model is split into stages, where each stage adds one of the techniques detailed in the previous subsection as follows:


These models will be referred to as PMSi, i.e., the Proposed Model at its Stage *i*. As in other works, for instance [11], the performance of these models has been analysed for every season of the year and compared with that of six other electricity price forecasting models, some of which correspond to well-established methodologies in the literature. The first chosen benchmark model (Benchmark 1 or

BM1) consists of the proposed simple average of [19] between the forecasts of a pure NN model and the base hybrid fundamental-econometric model (PMS0). Benchmark two (BM2) only involves this pure NN model that utilises the same input variables as BM1/PMS0 (except those pertaining to the fundamental model) and the same calibration window. This 120-day window includes four months within the 13-month window established in this work, more specifically, the 13th, 12th, 2nd and 1st month prior to the forecasting day [19].

The third benchmark model (BM3) is related to a linear regression model with several autoregressive terms and exogenous components. This ARX model, introduced in [32] and recently utilised in [15], includes a logarithmic transform that was modified so as to account for the lower price cap of zero in the Iberian electricity market:

$$\begin{split} p\_{d\boldsymbol{\beta}h} &= \beta\_{\boldsymbol{h},1} p\_{d-1,\boldsymbol{h}} + \beta\_{\boldsymbol{h},2} p\_{d-2,\boldsymbol{h}} + \beta\_{\boldsymbol{h},3} p\_{d-7,\boldsymbol{h}} + \beta\_{\boldsymbol{h},4} p\_{d-1}^{\text{min}} + \beta\_{\boldsymbol{h},5} \boldsymbol{z}\_{d,\boldsymbol{h}} \\ &+ \beta\_{\boldsymbol{h},6} D\_{\text{Sat}} + \beta\_{\boldsymbol{h},7} D\_{\text{Sun}} + \beta\_{\boldsymbol{h},8} D\_{\text{Mou}} + \boldsymbol{\varepsilon}\_{d,\boldsymbol{h}} \end{split} \tag{1}$$

$$m\_{d, \text{lt}} = \frac{\left(P\_{d, \text{lt}} - \mu\_T\right)}{\sigma\_T} \tag{2}$$

$$p\_{d,h} = \text{sgn}(n\_{d,h}) \left[ \log \left( n\_{d,h} + \frac{1}{c} \right) + \log(c) \right] \tag{3}$$

According to Equation (1), the log-price at day *d* and hour *h*, *pd*,*h*, depends upon: lagged prices, such as *pd*−1,*h*; the minimum log-price during the 24 h of day *<sup>d</sup>* minus one (i.e., *pmin d*−1 ); the load forecast (*zd*,*h*); and three dummy variables that specify if day *d* is a Saturday, Sunday or Monday. The logarithmic transform of Equation (3) is the mirror-log transform, where prices, *Pd*,*h*, are first normalised in Equation (2) across the training period *T*, and the parameter *c* was set to 1/3 according to the application presented in [33]. The transformations that have been applied to the explanatory variables regarding past electricity prices stabilise their variance and ensure stationarity, as observed in [33]. Three months prior to the forecasting day were used as calibration data. The next benchmark (BM4) is the extension of BM3 as per the work presented in [15], which performs a weighted average of forecasts from the ARX model of Equation (1) across the following calibration windows (in terms of days prior to the forecast day): 56, 84, 112, 714, 721 and 728 days. The weights of these six forecasts are computed by means of an inverse MAE weighting procedure when testing the ARX models on the day prior to the forecast day.

Benchmark five (BM5) is related to a SARIMAX model, whose SARIMA noise presents the following notation: SARIMA(1,0,0)168(1,0,2)24(1,0,0)1. A daily and weekly seasonality was considered, as well as the expected demand as an exogenous variable. This model was created following [34,35], with the Box-Jenkins methodology. Furthermore, the Box-Cox transformation was used to stabilise the price variance [36]. The final benchmark (BM6) is related to a simple naïve approach that sets the forecast to the actual electricity price value corresponding to the previous week.

The proposed model, in all of its stages as well as the six benchmark models, have been tested for every day of the year in 2017 and their error measures across the four seasons of 2017 are shown in Table 1. Furthermore, the average calibration dataset windows for each of the models involving a NN forecasting technique are displayed in Table 2.




**Table 2.** Average calibration length window of the NN models (days).

Compared with the base model of PMS0, the implementation of the structural breaks technique increased the NN training set by well beyond the predefined number of 120 days that was established in [19]. The reason behind the reduced dataset during the 2017 winter is due to its high instability, and it was observed in [19] that a reduction of the 120-day dataset provided useful results. This agrees with the rationale that consists of increasing adaptability on unstable periods by reducing the calibration window in order to remove structural breaks from the input dataset. However, in this work, an average dataset of 152.9 days yields lower forecasting errors. Furthermore, PMS1 discards most of the previous winter, which is considerably different from the 2017 winter as depicted in Figure 3. This also seems to be the case for spring, as the 2016 spring yielded approximately twice as much hydro generation as the 2017 spring. In general, the structural breaks algorithm provides a generally lower error throughout 2017. However, summer 2017 appears to be the exception, where prices are relatively stable and thus, it lacks room for improvement, as proven by the generally low errors yielded by most models.

Furthermore, the lengthening of PMS1's calibration dataset with the hourly clustering technique of PMS2 further reduces the overall forecasting error. This is more notable during winter, where the average calibration dataset is greatly increased to 288.8 days. As for the other seasons, a calibration dataset of approximately one year proves to be beneficial for electricity price forecasting with NN models even with the hourly arrangement and does not seem to cause any overfitting issues. Although PMS2 yields a lower error overall, the statistical significance of these error measures must be verified in order to confirm its superiority against its competitors, especially the highest-ranked models according to Table 1. Therefore, a DM test was carried out for PMS2 against every other model. The DM test statistic is evaluated with a 5% significance level, such that a DM statistic < −1.96 implies significant outperformance. The results of the DM test statistic are shown in Table 3.


**Table 3.** DM statistic values of PMS2 against every other model.

The three values in bold indicate the three occasions that PMS2 was unable to significantly outperform. The comparison with PMS0 suggests that the increase in calibration data window lengths does not significantly contribute to summer forecasts, albeit not detrimental to the accuracy. This may also imply that a robust calibration period selection is not highly crucial in such a stable market regime. Therefore, the same conclusion can be drawn from the summer comparison with PMS1. Furthermore, the DM statistic value in autumn when compared with PMS1 may indicate that the information provided by the hourly clustering method is not significantly different than that provided by the structural breaks technique. However, these values indicate that PMS2 is significantly outperforming all other models throughout the year in 2017.

#### **4. Conclusions**

This research presents a novel short-term hybrid electricity price forecasting methodology which is comprised of three main elements: a cost-production optimisation model, a sophisticated period filtering approach and a neural network (NN) model. These three elements were utilised sequentially with the calibration selection procedure as the main focus of this work. Given a forecasting day, the structural patterns in actual prices corresponding to the 13 months prior to that day are analysed and those deemed unimportant were discarded. A K-means clustering method was also applied to relate the moments in the prior 13 months to the forecasting day in terms of the estimated fundamental market-clearing prices, expected thermal gap and expected mean temperatures in the Iberian Peninsula. The key innovation of this approach is to move beyond the conventional forecasting principles which suggest that adaptive methods should place most emphasis upon recent data and that regime-switching should likewise model transitions from the latest regime. The approach recognises that the most relevant dataset in the episodic, recurrent nature of electricity dynamics may not be the most recent. Another unique feature of this methodology is the definition of a calibration period that is not driven by heuristic assumptions or any other specific predefinitions.

The results and analyses indicate the following. The combination of structural break analysis and hourly clustering provides a dynamic calibration period appropriate for the forecasting model estimation. In validation, this sophisticated training window selection for the NN model yields appealing results in every market circumstance present in the relatively challenging case study of the Iberian electricity market of 2017. The period selection technique is more selective in volatile market conditions, such as early 2017, albeit providing a considerably longer training window length than other works which claim that employing much shorter calibration windows is most suitable in these situations. In addition, the proposed methodology proves most useful during volatile periods, whilst the accuracy is marginally increased in stable market regimes, such as summer 2017.

Overall, this short-term fundamental-econometric electricity price forecasting model, which features a unique hybridisation approach, has yielded appropriate results when applied to a real-size electricity system with complex price dynamics, such as the Iberian power exchange of 2017. Furthermore, the performance of this proposal is superior to that of other benchmark models. Although only one market has been chosen as the case study, the results may be generalised for other markets due to the high number of special circumstances that the Iberian power system experienced throughout the year 2017. However, there seems to be room for improvement regarding the utilised structural breaks period selection algorithm, as it is highly challenging to ascertain a convenient compromise between accuracy and computational burden. Transient spikes for example cannot all be considered structural breaks, yet it may be beneficial if these are more adequately considered in a computationally feasible manner. Furthermore, more complex neural network topologies may be tested in conjunction with this calibration period selection methodology, such as convoluted or LSTM neural networks.

**Author Contributions:** Conceptualization, R.A.d.M., D.W.B., A.B. and J.R.; Data curation, R.A.d.M.; Formal analysis, R.A.d.M.; Investigation, R.A.d.M.; Methodology, R.A.d.M., D.W.B., A.B. and J.R.; Software, R.A.d.M.; Supervision, D.W.B., A.B. and J.R.; Validation, R.A.d.M. and A.B.; Visualization, R.A.d.M.; Writing—original draft, R.A.d.M., D.W.B. and A.B.; Writing—review & editing, R.A.d.M., D.W.B., A.B. and J.R. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Energies* Editorial Office E-mail: energies@mdpi.com www.mdpi.com/journal/energies

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34 Fax: +41 61 302 89 18