Data-Centric Communication Framework for Multicast IEC 61850 Routable GOOSE Messages over the WAN in Modern Power Systems

Abstract

In this paper, a data-centric communication framework is proposed for multicast routable generic object-oriented substation event (GOOSE) messages (MRGM) over the wide area network (WAN) for effective substation-to-substation (SS2SS) and substation to control center (SS2CC) communications. In this structure, the IEC 61850 GOOSE message is transmitted over the WAN using the data distribution service (DDS) as a fast, reliable, and secure data-centric communication middleware. The main feature of this framework is its multicast capability, where several authorized subscribers can receive a published message simultaneously. This can significantly improve the system monitoring and control of the protection systems in modern smart grids, where intelligent schemes can be applied. The effectiveness of the proposed platform, in terms of total end-to-end delay between participants, is evaluated through experimental results obtained from the actual hardware-based test setup developed at the Florida International University (FIU) smart grid testbed. The results demonstrate that the latency between sending and receiving a GOOSE message among participants is within its maximum time span defined by the IEC 61850-90-5 working group for communications over the WAN.

1. Introduction

The substation automation system (SAS) plays a vital role in the optimal and reliable operation of modern power systems. The SAS is used for automating the control systems of substations and developing the remote monitoring and control mechanisms in energy industry [1]. Furthermore, it defines protocols for critical communications among Intelligent Electronic Devices (IEDs), which should be fast, reliable, and secure enough to address the power system operation, control, optimization, and protection issues. These all bring a new level of challenges to the power system control and protection.

Generally, communication protocols and data model for data exchange among the IEDs are defined through substation automation standards. Although a variety of standards, protocols, and technologies have been developed in this regard, many of them are vendor dependent and they do not properly address the interoperability issues in complex systems, and also do not support high speed communication technologies (e.g., Ethernet). Among all these standards, IEC 60870, MODBUS, and DNP3 are still being used in industry [2]. The IEC 61850 standard is the newest one which tries to address the above-mentioned drawbacks through new data models and protocols [3].

The IEC TC57 developed the IEC 60870 for basic remote-control communications between remote units and master stations and defined the profiles for different vendors to address the compatibility issues [4]. On the basis of the open systems interconnection (OSI) model [5], which categorizes the communication network into seven layers, this standard mainly acts on the data link layer and uses RS-232, RS-485, or fiber optic interfaces at the physical layer for point to point network topology communications. Nowadays, the MODBUS protocol is used widely in industrial applications because it supports different network technologies such as serial communications, optical/radio networks, RS-232, RS-485, and RS-422. The MODBUS was basically developed for response/answer manner and acts at different layers of the OSI model [6]. The MODBUS is optimal protocol for serial communications and originally was developed for master-slave communication manner which means that it inherently is not suitable for Ethernet communications. Although the MODBUS plus covered the internetworking remote communications over the internet by adopting the TCP/IP protocol, this protocol still suffers from some disadvantages. For example, the sequence of events can be missed due to a lack of time stamping action. The distributed network protocol (DNP3) is usually used within the SCADA systems and defines the rules for interconnecting the automation systems [7]. Considering the OSI model, the DNP3 is a Layer-2 protocol with related addressing mechanism, link control, multiplexing, etc. Although the DNP3 is highly used in power industry applications, its main drawback occurs in the interconnection with data units and substation transport events, where the DNP3 protocol data packets lose their logical context.

In response to the above-mentioned standard limits, the IEC 61850 standard was developed to implement the modern networking technologies, data model, and communication protocols [8,9,10]. This is a comprehensive standard with advanced protocols for device integration, data encapsulation, and network services, and therefore it provides a highly flexible design environment for each project and considers the communication technologies, object definition, and requirements. For example, there are three defined data models in this standard for different applications, i.e., GOOSE, sampled measured values (SMV), and manufacturing message specification (MMS). The GOOSE stands for the generic object-oriented substation event and is used for critical, time-sensitive, and multicast applications such as tripping/blocking signals within the wall of a substation. This type of communication occurs over the data layer (Layer 2) of the OSI model and must be completed in less than 4 milliseconds (ms). The SMV and MMS data models are used for sampled measured values and manufacturing message specification, respectively. In addition to communications over the data layer (horizontal communications), the routable communication protocols (vertical communications), such as routable GOOSE (R-GOOSE), for internetworking communications over the network layer (Internet Protocol (IP) layer) has been defined by IEC 61850-90-5. This communication is also time sensitive and should be highly reliable and completed over a 10 ms time span including the network topology convergence. However, communication over the network layer using the Internet Protocol (IP) deals with many limits because they were not built for these types of communications which are highly sensitive in terms of reliability, delivery order, and communication latency.

The R-GOOSE message, along with its applications in distribution automation, were described in detail in [11], where authors demonstrated the difference between GOOSE and R-GOOSE message and their data models and features. However, a proper communication middleware was not introduced, and network issues were not addressed. The applications of DDS in smart grids were evaluated in [12]. For example, DDS was used to improve smart grids communications by addressing the interoperability between different protocols such as DNP3 and MODBUS. However, it did not consider and address interoperability issued related to the IEC 61850 and routable communications. To route a GOOSE message over the WAN, adaptors are needed for configuration and interface with various equipment which use their own communication protocols such as Modbus, DNP3, IEC 61850, etc. For this purpose, by considering the formats which are described by data profiles, adaptors convert data and send them over MQTT, DDS, or AMQP communication protocols [13]. In this reference, the GOOSE message is translated to the OPC by the user agent (UA) to be transmitted over the WAN. However, this reference did not map the GOOSE data model over a DDS data object. In [14], a mechanism was introduced to deploy the IEC 61850 routable sample values (R-SV) and routable GOOSE (R-GGOSE) messages for reliable data transfer of Phasor measurement units (PMUs) over the WAN to facilitate the communication for wide-area monitoring, protection and control (WAMPAC). But the application of a broker-less communication middleware for fast and reliable data delivery was not presented. In [15], the DDS was deployed as a communication backbone for SAS. The process of mapping different communication traffics (GSE, SMV) into DDS data object were described properly. However, network issues, routing service, and end-to-end delay for routable communication over the WAN were not addressed. For a successful routable communication over the WAN that meets the maximum acceptable latency, network traffic management is needed. Usually, tunneling techniques could help to manage traffics between networks. Nowadays, by creating the 5G networks, a specific slice can be allocated for delay sensitive messages and, for example, slice isolation can help with managing the traffic as mentioned in [16]. This reference categorizes the different message types used in distribution grid protection and control from the standard defined by IEC 61850 and maps onto the three major groups of 5G use cases. It is worth mentioning that the 5G can address the end-to-end delay challenge in the substation-to-substation communication, however utilizing DDS in addition to the 5G can get the maximum benefit of the 5G network. DDS provides low latency scalable communication middleware, in addition to flexible and extensive Quality of service (QoS) profiles which can be applied to individual data types instead of applying QoS to the whole stream or protocol in addition to QoS profiles, DDS binary encoding, and data filtering reduce the network bandwidth. Mapping GOOSE messages to DDS provides a solution that can be deployed using current technology and migrates it easily to the 5G when it becomes available. Moreover, due to possible cyber-attacks, the communication network should be isolated properly. One solution is to use the virtual private network (VPN) for encryption and isolating data. However, to overcome the routable communication requirements, the network enhancement through special technologies is necessary.

The contribution of this paper is to address the above-mentioned weaknesses for fast, multicast, and reliable IEC 61850 R-GOOSE communications over WAN in modern power systems. Therefore, the state-of-the-art of this work could be summarized as follow:

• Introducing a DDS-based communication framework for multicast R-GOOSE messages;
• Describing unique features of this framework in terms of scalability, reliability, and multicasting;
• Addressing the network traffic issues and minimize the end-to-end time delay;
• Verifying the proposed approach through an experimental setup by measuring the real-time end-to-end delay of R-GOOSE message communication over the network layer.

In other words, we are going to address this challenge for IEC 61850 routable communications by introducing a data-centric communication framework for R-GOOSE messages using the data distribution service standard. The proposed solution is a fast and reliable framework that covers all the IEC 61850 routable communication requirements along with the feature of multicasting, which could enable us to develop advanced protection schemes for modern power systems. Furthermore, we develop an experimental setup to measure the real-time end-to-end delay of R-GOOSE message communication over the network layer to verify the effectiveness of the proposed framework.

The rest of this paper is as follow: In Section 2, the message centric and data-centric communication approaches are described and compared. Section 3 is devoted to present the data distribution service as a data-centric communication standard. In Section 4, the proposed multicast routable GOOSE messages (MRGM) communication framework is presented in detail. The proposed test framework along with hardware setup and experimental results are reported in Section 5, and finally, the paper is concluded in Section 6.

2. Message-Centric vs. Data-Centric Communication Approaches

The message-centric communication approach is used by traditional communication middleware, where a set of messages and data formats are predefined in participants’ nodes. In this approach, the communication middleware only delivers the message through the network layer and does not know the message contents and data types. Messages are being appraised within each node of the system (at the application layer) to check the correctness, integrity, data type, and filter data. It means that each node should locally track the state of data, which significantly increases the processing tasks on the application layer. Furthermore, the system expansion is very complicated because major changes are required on the application layer due to any change in message formats or data types [17,18].

Nowadays, modern communication middleware uses the data-centric communication approach. In this data approach, the communication middleware builds the message and updates the system state, therefore, it is aware of the message content and data model [19]. Moreover, instead of processing the messages locally in application, as is done in a message-centric approach; in a data-centric approach the messages are being processed within the middleware layer to assess the correctness of data types delivered to all nodes. It simplifies the application development and system expansion, enhances the system reliability, optimizes the use of network bandwidth, and gives us more opportunities to assign different Quality of Service (QoS) profiles, priorities, and security measures to the data types as comparing with a message-centric approach [18]. In this paper, the DDS is used as a data-centric communication middleware to address the challenges of routing GOOSE message over the WAN. In the following section, DDS is described, and its unique features are mentioned in detail.

3. Data Distribution Service

The data distribution service (DDS) is a data-centric communication standard which was initially released by the Object Management Group (OMG) in 2004 [20]. The DDS deploys a data-centric publish-subscribe (DCPS) protocol within a virtual global data space to facilitate the communication among participants [21,22,23]. The DCPS protocol is based on defining topics with specific data types and formats through IDL (interface definition language) files over a DDS domain. Afterwards, participants with data writer can publish into these topics and subscribers can use data reader to receive data. It is worth mentioning that several subscribers are able to receive data simultaneously once data is written or updated on a topic, therefore, it is a multicast communication framework which facilitates the data exchange among participants with the same domain ID [15]. This communication platform is fast, expandable, and reliable enough and can be utilized in critical real-time applications. The automatic discovery feature of the DDS makes it possible for dynamic participants to be added/deleted to/from the system without any interruption in the system operation [24,25,26]. The DDS is also equipped with an extraordinary governance and management tool, called Quality of Service (QoS), for flexible communication and controlling the system behavior in terms of data latency budget, priority order, lifespan, durability, etc.

The single point of failure feature in most of the message centric communication middleware directly affect the reliability of system. However, the DDS does not need a message broker for peer-to-peer communication, and therefore it is an excellent and highly reliable tool for distributed applications. In addition to high reliability of the DDS, it is easily scalable due to the real-time publish-subscribe (RTPS) protocol, automatic discovery (dynamic participation of network nodes), and routing service for distributed applications with participants in different networks and diverse transport protocols (e.g., IPv4, IPv6, or shared memory) [27,28,29]. In fact, tracking millions of data values and self-forming of autonomous systems are possible because of scalability and automatic discovery features, respectively. The DDS is optimized to provide a reliable, scalable, and secure integration bus to connect existing and new assets. In dynamic networks, the applicants are plug-and-play if DDS is deployed, and therefore there is no need to deploy configuration because of automatic discovery ability of this middleware. Although DDS could be expanded significantly, its performance is still highly efficient. For example, increasing the number of subscribers from one to 888 only has an impact of 10% on throughput. In addition, the DDS can be implemented for ultra-large-scale systems (ULS) using DDS-wire protocol which is known as DDSI for scalability on ULS deplanements over WAN [28]. The DDS routing service is also achievable through the QoS profile, where the IP address of participants along with proper network protocol (e.g., TCP or UDP) is defined. It results in routing the published data from the first network (where the publisher is located) to the predefined domain ID and topic name at the subscriber’s network.

The security of the DDS is another unique feature of this standard. It allows participant to be authenticated before participating in the communication procedure to avoid system spoofing by unauthorized agents, provides an encryption/decryption mechanism through a key management mechanism for secure data exchange among participants to avid data spoofing and bad data injection, and defines a redundant security measure called permission access control, which determines the accessibility for each participant to each domain, topic, and data along with the right to write or read data from that topic. This permission accesses are evaluated through a permission certificate authority, which is initially responsible for signing the certificate for each participant.

4. Proposed MRGM Framework

Figure 1 shows the proposed DDS-based communication framework for multicast R-GOOSE message (MRGM) over the WAN. It also contains two algorisms for GOOSE message transition from the first substation (Substation A) to the second one (Substation B) which are assumed to be in two different local area networks (LAN). As can be seen in this figure, the DDS is selected as a data-centric communication middleware for GOOSE data exchange between two participants in two substations. The IEC 61850 defines a real-time publish-subscribe (RTPS) protocol for GOOSE data exchange within the wall of a substation. In this protocol, the GOOSE message is published by the GOOSE publisher to the local network with specific format including sequence number (SqNum), status number (StNUm), GOOSE dataField, and an APPID (It is the ID of the published message which is checked by subscribers). Once the message is published, subscribers with the same APPID instantly subscribe to the message and praise the message to take an appropriate action. The rate of sending GOOSE messages varies due to the system condition. In normal operating mode, where there is no need for protection relays’ actions, GOOSE messages are sent periodically to test the communication system. It usually happens every one second.

The high-frequency cascading messages are sent due to protection action requirement because of fault detection in the system or control commands. In this situation, as shown in Figure 2, a train of GOOSE messages are sent to the network starting with a very high frequency and continue to lower frequency rates to make sure that the messages are received by appropriate subscribers to cover the possibility of missing samples. The standard has defined a 4 ms time span for GOOSE message delivery within the data link (Layer 2 of the OSI model). However, the published GOOSE messages could be needed to be subscribed by other IEDs in other substations or by the control/monitoring center. For this purpose, GOOSE messages should be transferred from the data layer (Layer 2 of OSI model) to the network layer (Layer 3 of OSI model) and routed to their destinations in the system over the internet network. The standard has defined a 10 ms time span for routable GOOSE (R-GOOSE) messages.

As mentioned before, specific network settings are required to address this fast delivery over the WAN, therefore, in this paper we propose to deploy the DDS standard for R-GOOSE messages as it covers all the required communication needs. Furthermore, it provides the multicast feature of R-GOOSE messages at the network layer, the same as GOOSE messages at the data layer. This means that several subscribers could receive a published GOOSE message simultaneously and take proper actions immediately. This feature is very helpful in advanced protection systems. In [30], authors have presented an advanced protection scheme for a modern power system, which are experiencing a high level of fault current beyond the breaking capacity of circuit breakers. The proposed advanced protection scheme is mainly based on R-GOOSE messages over the WAN and assumes that a multicast R-GOOSE message framework could deliver the messages to its destination in less than 10 ms. This paper is addressing this framework and demonstrating its capabilities through experimental results. The proposed MRGM algorithm includes three main steps starting with GOOSE publisher in the first network, routing the message over the WAN, and subscribing it within the second network.

4.1. Conversion: GOOSE Data Model to DDS Data Object

As shown in the right side of Figure 2 (Substation A), the R-GOOSE communication is begun similar to the conventional GOOSE publishing in data layer. Since GOOSE data model has been defined in the IED platform, to change and add or modify the feature of message we need to develop agents with access to the same network, where GOOSE message is published. This agent is equipped with a GOOSE subscriber with the same APPID of the GOOSE message that we need to send it outside the wall of this substation. Once the GOOSE message is subscribed by this agent, the algorithm is activated by praising the received message, extracting its parameters, and activating a DDS publisher. As mentioned before, a GOOSE data model contains important parameters, i.e., StNum, SqNum, and DataField. For broadcasting these parameters by a DDS publisher, a proper IDL file should be defined by suitable data format for each parameter. The StNum and SqNUm contain integer values while DataField has the Boolean type. Therefore, these data types are defined on a DDS topic (e.g., Topic1) within a DDS domain (e.g., Domain n) by three variables (A, B, and C). The activated DDS publisher writes the received parameters’ updates on this topic.

4.2. Message Encapsulating and Routing Service

To encapsulate this message and route it over the WAN, the DDS routing service is activated and simply routes this message over the network router and delivers it to a predefined topic within a DDS domain in another network, where DDS subscribes are waiting to read data. This is possible just by defining the IP address of both DDS publisher and subscriber through the DDS routing service profile.

As mentioned before, to meet the maximum end-to-end delay for R-GOOSE communication, specific network settings are needed. The DDS QoS profile contains control mechanism and policies for data transfer over the network. For example, it could control availability of data, life span, latency budget, etc. [31]. The message delivery time is defined by the latency budget, i.e., the index which shows the maximum allowed time span for the message delivery. To control this latency, message priority indices could be attached to time-sensitive and critical messages, therefore, the network switches transfer the received messages based on the priority policies, regardless of data queue. This means that in limited bandwidth networks with high message deliver latencies, which usually is due to the first come first serve policy, defining the message priority profiles could significantly enhance the critical message communications over the internet network [32]. The DDS supports three priority policies including round ronin (RR), earliest deadline first (EDF), and high priority first (HPF). The RR is known as the simplest priority scheduling scheme which processes the messages in the order they have been received. It prioritizes the messages considering their latency budget scheduled in EDF policy, where the messages with less remaining time span are processed first. It is worth mentioning that the EDF is the default policy scheduled by the DDS. The combination of the RR and EDF could be scheduled by HPF policy, where the messages with the same latency budget are processed by the RR policy [31].

4.3. Conversion: DDS Data Object to GOOSE Data Model

By delivering the message to its destination in the Substation B’ network, a predefined topic (e.g., Topic2) within a domain (e.g., Domain k), a DDS subscriber deploys its data reader to receive data, as shown in the Figure 2, Substation B’ algorithm. Afterward, these data are tuned within a GOOSE publisher and published to the local network over the data link layer. Finally, actual IEDs with the same APPID subscribe the message, verify, and react to it.

The same algorithm is run with all other substations or monitoring centers which are supposed to receive data from this GOOSE publisher. The multicast feature and simultaneous subscribing of the R-GOOSE message enable us to implement real-time, reliable, and advanced protection, control, or remedial actions in modern smart power systems [30].

5. V. Test Framework

To evaluate the performance of this framework in terms of data delivery and delay time, we proposed a test framework, as shown in Figure 3. The purpose is to measure the latency of the algorithm for both GOOSE publishing and subscribing in the data link layer and the DDS time delay over the network layer. This framework contains four Linux-based embedded agents shown by numbers (1) to (4). Agent (1), receives the analog input signal, converts it to digital mode (0/1), sets the GOOSE DataField according to this value, and publishes it to the network by the APPID # 1000. Agent (2) has an active GOOSE subscriber with the same APPID (#1000) and subscribes to this message. The GOOSE DataField along with StNum and SqNum are extracted and are used by DDS data writer to publish these data over the DDS topic. In Agent (3), the DDS message is subscribed and being published by the GOOSE publisher with another APPID (# 2000). Agent (4) receives this message, converts its digital DataField value to an analog signal, and delivers it to output ports of this agent, where this signal could be compared with input to determine the time delay due to communication platform.

5.1. Hardware Setup

According to Figure 3, we developed a hardware-based experimental setup at FIU smart grid testbed, shown in Figure 4. In this setup, we used four Beaglebone Blacks as defined agents for this communication framework (each one has an AM335x Arm^® Cortex-A8 with 1 GHz processor and 512 MB RAM), a function generator to generate the input signal with 3.2 volte amplitude and frequency of 40 Hz, and an oscilloscope to monitor and compare the input and output signals. A personal computer (PC) was deployed to monitor the RTI admin console, where we can monitor the DDS system participants including the publishers and subscribers, DDS domains, and topics.

5.2. Experimental Results

5.2.1. Case (1) GOOSE Publish-Subscribe over Layer Two

This experiment is executed using two agents, as shown in Figure 5. To evaluate the latency of the GOOSE delivery, we needed to examine a large number of samples (around 1k) because of various time delays between input and output signals. It is because of this fact that the processors deal with each sample differently, therefore, we had calculate the average latency of GOOSE communication over the data link layer. The minimum and maximum latency recorded in our measurements were around 300 and 1300 μs, respectively, and the average value was around 0.5 ms (TG = 500 μs) with a standard deviation (S.D.) of 50 μs. The low value of the standard deviation indicates that the average value is a good index to demonstrate the time latency of GOOSE communication at data link layer.

5.2.2. Case (2) DDS Publish-Subscribe over the Network Layer

To measure the DDS delay, we had to measure the total end-to-end delay (Tt) by the proposed test framework, as shown in Figure 6. As mentioned before, this communication occurred by two GOOSE publish-subscribe actions plus a DDS data exchange. Therefore, by measuring the total end-to-end delay, the DDS delay time (TD) is calculated by (1).

$$T_D=T_t-\left(2\times T_G\right)$$

(1)

The average value of the total end-to-end delay (Tt) was obtained by recording a large number of samples (around 1k) as we did in Case 1. The latency of this case is changing between 1000 and 2700 μs. The average and standard deviation are 2100 and 160 μs, respectively. As a result, and as shown by (2), the DDS delay time is 1100 μs.

Table 1. Results from Case 1 and Case 2.

Study Case	Latency (μs)
	Min.	Max.	Ave.	S.D.
1	300	1300	500	50
2	1000	2700	2100	160

summarizes the results for these two study cases.

$$T_D=2100-\left(2\times 500\right)=1100 \mu s$$

(2)

The test framework does not consider the latency of the DDS routing service and network traffic. However, experimental results report an average of around 70 μs for the DDS routing service action (TR). Hence, the average latency due to network traffic (TN) should not be more than 7830 μs, as calculated by (3).

$$T_N=T_{Max}-\left(2\times T_G+T_D+T_R\right)$$

(3)

In this equation, T_Max is the maximum allowed time delay for R-GOOSE messages, which is 10 ms, based on IEC 61850 standard. The average time delay for each section of proposed communication framework is reported in

Table 2. Average latency for each section of the proposed framework.

Section	Latency (μs)
$T_G$	500
$T_D$	1100
$T_R$	70
$T_N$	7830

. These calculations mean that the proposed framework is fast enough and only needs around 2170 μs to encapsulate a GOOSE message and transfer it by the DDS and give us 7830 μs for communication over the WAN, which is feasible by defining a tunnel between two communicating networks along with a proper message priority policy. Generally, the network latency is highly dependent on the size of the message, network bandwidth, and performance (unicast/multicast features). As reported in [31], for a 32 bytes message size with the rate of 1 k message per second, the best effort of the DDS QoS profile resulted in 269 μs latency for the communication system. It means that using a proper QoS profile and tunneling technique, the R-GOOSE communication would be feasible over the maximum time span (7830 μs) available when the proposed framework is utilized.

6. Conclusions

In this paper, a multicast communication framework was proposed for R-GOOSE messages over the WAN. The skeleton of this framework is based on a data-centric communication approach to address the technical requirements for time sensitive and critical communications in smart power systems. This approach encapsulates the GOOSE data model into the DDS data object and routes it over the network to its final destinations. The effectiveness of the proposed framework was validated by the experimental results by measuring the latency of this framework, showing that this approach only took around 20% of the maximum defined time for IEC 61850 routable communication and give us around 8 ms to manage the network traffic. Furthermore, the best effort of the QoS profile along with the proper message priority policy could guarantee the message delivery within its defined time span.