Tunneling Horizontal IEC 61850 Traffic through Audio Video Bridging Streams for Flexible Microgrid Control and Protection

Abstract

In this paper, it is argued that some low-level aspects of the usual IEC 61850 mapping to Ethernet are not well suited to microgrids due to their dynamic nature and geographical distribution as compared to substations. It is proposed that the integration of IEEE time-sensitive networking (TSN) concepts (which are currently implemented as audio video bridging (AVB) technologies) within an IEC 61850 / Manufacturing Message Specification framework provides a flexible and reconfigurable platform capable of overcoming such issues. A prototype test platform and bump-in-the-wire device for tunneling horizontal traffic through AVB are described. Experimental results are presented for sending IEC 61850 GOOSE (generic object oriented substation events) and SV (sampled values) messages through AVB tunnels. The obtained results verify that IEC 61850 event and sampled data may be reliably transported within the proposed framework with very low latency, even over a congested network. It is argued that since AVB streams can be flexibly configured from one or more central locations, and bandwidth reserved for their data ensuring predictability of delivery, this gives a solution which seems significantly more reliable than a pure MMS-based solution.

1. Introduction

Traditionally, for economic and safety reasons, electricity was generated by large centralized fossil-fueled generators, and transported to consumers via one-way transmission networks, load centers, and distribution networks [1,2,3]. However, the liberalization of the energy markets, combined with environmental, sustainability, and reliability concerns, has forced a rethink in the way that electricity is generated and distributed to consumers [4]. Specifically, recent times have seen the emergence of small- and medium-scaled distributed energy resources (DERs) embedded within the transmission and distribution networks, typically in the form of renewable/sustainable energy generation sources such as biomass-fueled combined heat and power (CHP) plants, wind turbines (WTs), and photovoltaics (PVs). Although such technology has the potential to improve reliability and flexibility of the network, without adequate supervisory control, monitoring and reactive supply/demand side management, problems related to grid frequency/voltage control combined with loss of stability and activation of protection systems may arise [4,5].

This has led to the development of the concept of the smart grid, i.e. an energy distribution network that not only allows for the physical transfer of energy but which also supports information and communication technology (ICT) interfaces enabling real-time information exchange related to the scheduling, monitoring, control, and protection of the interconnected DERs [4,5,6]. A closely related concept to the smart grid is the microgrid. A microgrid is a modern, small-scale instantiation of the larger centralized electricity generation, transmission, and distribution system [7,8,9]. A microgrid generates, distributes, and regulates the flow of electricity from generators to consumers on a small (local) scale. A microgrid has an electrical point of common connection (PCC) with the larger grid, and may operate in “grid connected” mode or “islanded” mode, dynamically switching between the two depending upon internal and external operating conditions (the latter operating mode being the most common). A microgrid has local targets and constraints (such as reliability, emissions reductions, DER integration, and economic cost minimization), but also interacts with the wider smart grid to help achieve larger-scale targets. When employed correctly in co-ordination with the wider smart grid, microgrids may be used as “good citizens” which can provide adjustable and predictable power sourcing/sinking. This can aid in load balancing of the wider network and an increase in network reliability and security [6,7]. Microgrids can simplify problems related to the complexity of wider network control and management as system of systems (SoS) concepts can be applied [9]. A high-level view of a typical microgrid and its constituent components is shown in Figure 1 below.

As can be seen in Figure 1, a number of dispatchable and non-dispatchable generation units (CHP, WT, PV, etc.) are electrically connected to load units (domestic, commercial, small-scale industrial, etc.) through a private wire electrical network. Other units may be present in this network, such as electric vehicle charge points (EVCP) and battery energy storage systems (BESSs). The microgrid connects to the wider grid via the PCC. Each unit is typically connected to the electrical network through a disconnection (no-load) switch and protective circuit breaker. Each unit is controlled by a local intelligent electronic device (IED), and each IED is interconnected via a communication network in which control, status, and monitoring information pertinent to the microgrid operation is exchanged. In a centralized microgrid control approach (which is most commonly used) a control center (CC) is also present and connected to each IED via the communications network. The CC is responsible for coordinating the microgrid as a whole, and it typically plans, optimizes, and manages the medium-term and short-term operating strategies of the network and also helps to facilitate real-time control, monitoring, and protection. The medium-term strategy is often determined by solving an economic dispatch optimization problem over a future horizon of approximately 24 h (e.g., see [8,9]), typically using an hourly timescale. The short-term strategy typically helps to put this hourly dispatch plan into practical action and operates on a typical time scale of 1–5 min [9]. Supervisory control and data acquisition (SCADA) concepts are often used at this timescale, for example to read a BESS state of charge and set its rate of charge/discharge. At the very lowest timescales, the CC helps to facilitate real-time control, monitoring, and protection functions in the microgrid [8,9]. Timescales at this level are typically in the sub-second range and distributed control system (DCS) concepts are often used alongside SCADA, for example to schedule dispatchable DERs to regulate microgrid power exchange over the PCC or frequency.

In this paper, the focus is upon communication infrastructures and technologies for command, control, monitoring, and protection in microgrids. The paper considers microgrids and small smart grids operating at the “Neighborhood” level, in which it is implied that the communications between DERs take place over a distance of not more than approximately 20 km. Within such a framework, the IEC 61850 standard for communications within substations—and its extension to DER management—is seeing increasing use [7]. Along with the standard mapping of IEC 61850 to Ethernet and manufacturing message specification (MMS) using transmission control protocol and internet protocol (TCP/IP), this is becoming the preferred means to interconnect and operate a diverse range of microprocessor-based electrical control and protection equipment. It is argued in this paper, however, that some low-level aspects of IEC 61850 are not especially well-suited for use with microgrids. In microgrids, there are typically requirements for node discovery and regular re-configuration of operations (e.g., enforced by DER reliance on weather conditions), coupled with the possibility of widely spread geographical areas and the need for command and control traffic to possibly co-exist and share resources with regular internet traffic. Such features are not typically present in substations which have comparatively smaller fixed boundaries and fixed equipment, plus isolated/firewalled internal communication infrastructures. In particular, the mapping of IEC 61850 horizontal communications to raw Ethernet does not seem well suited for reliable and timely communication between microgrid DERs. In this paper, it is proposed that the integration of IEEE time-sensitive networking (TSN) concepts (which are currently implemented as audio video bridging (AVB) technologies) within an IEC 61850 and MMS/TCP/IP framework provides a flexible and reconfigurable platform capable of overcoming such issues. By mapping horizontal protection and control messages between DERs into AVB transport protocol (AVB-TP) frames, the required time-critical information flows for protection and control may be carried over a much larger geographical area than a typical substation using low-latency AVB streams. AVB streams can be flexibly configured from one or more central locations, and bandwidth reserved for their data ensuring predictability of delivery even when mixed with regular internet traffic. A prototype test platform is described in this paper and it is verified that IEC 61850 event and sampled data may be reliably transported within the proposed reconfigurable framework, even in the presence of congestion.

The remainder of the paper is organized as follows. Section 2 presents a brief background on smart grid and substation communications and describes the IEC 61850 standards. Section 3 discusses IEC 61850 DER extensions, and outlines potential difficulties with regard to microgrids. Section 4 describes the AVB standards and presents a methodology whereby critical control and protection traffic may be tunneled through a current-generation AVB network. Section 5 details a prototype implementation and initial experimental tests which validate the concepts. Section 6 concludes the paper and suggests areas for future work.

2. IEC 61850 Substation Communications

The IEC 61850 standard [10] was principally defined to specify information models, naming conventions, communication principles, and communication service requirements to be employed in substation automation systems (SASs) employing microprocessor-based intelligent control and monitoring systems. It has since been extended and appended with numerous sub-standards which specify information models for automation and communications within and between DERs, WTs and HydroPower plants, among others [5,7,10]. In this Section, an overview of relevant aspects of IEC 61850 is first presented, followed by microgrid-specific discussions in Section 3.

2.1. The Main IEC 61850 Standards

The purpose of the main standard is to describe common application level features (object models, data semantics) which allow heterogeneous IEDs to interoperate and communicate efficiently within a SAS. The standard is also concerned with classification of common SAS communications services classes and their requirements, but not with the lower-level implementation details of a specific protocol stack. The standard principally supports the operation, control, and automation of substation equipment, and also specifies facilities for monitoring and control by remote control stations. The IEC 61850 information model defines naming and addressing schemes for a distributed database virtualizing a SAS. The database consists of data objects grouped according to their Data Class and contained within logical nodes (LNs) which are virtualizations of physical objects or functions. One or more LNs are contained within a logical device, which is encapsulated within a physical device having a concrete network address. Metadata (attributes) related to a particular data object (or objects) are also held by LNs and logical devices (LDs). Within this model, the topology of a substation can be divided into three levels: the Station level, the Bay (or Unit) level, and the Process Level. The power system and field interconnect devices sit below the process level. Extensions to the main standards provide a means by which the abstract models and services defined in the main standard may be mapped to actual communication protocols and technologies. IEC 61850 communications within substations are principally based on a mapping to the Ethernet protocol using store-and-forward switches operating at bit rates of 100 Mbps. Multiple bridged substation local area networks (LANs) are normally identifiable within the substation: the station LAN at the Station level along with (possibly multiple) process LANs at the Process Level, with the presence of field communication networks such as controller area network (CAN) and serial (RS-232) links possibly sitting below or as a part of the process level. The general layout of components and interconnecting LANs in a substation automation application is as shown in Figure 2.

Typically, microprocessor-based analog merging units (AMUs) and input output units (IOUs) are located in each Bay. AMUs provide connectivity to one or more field devices producing raw analog sensor data (from devices such as current transformers (CTs) and voltage transformers (VTs)), concatenate these signals and provide Ethernet connectivity of these devices to the Process LAN. IOUs provide connectivity to one or more field devices producing and consuming raw digital inputs and outputs (such as circuit breakers (CBs) and disconnect switches (DS)), concatenate these signals and again provide Ethernet connectivity to a Process LAN. Field communication networks may often be used to connect devices to AMUs and IOUs. A process LAN conveys the raw power system information (merged CT and VT samples, CB status and control bits) from these AMUs and IOUs to microprocessor-based protection and control relays (the IEDs) in each Bay. The IEDs represent the logical devices within the information model, with individual protection and control functions forming the LNs. The IEDs process the raw data into useable physical measurements and make decisions on the required switching devices’ statuses based upon these processed measurements and the internal IED protection/control logic and parameter settings.

The decisions made by the IEDs (required statuses for the switching and control equipment) are then conveyed back to the relevant CBs and DSs via the process LAN and through IOUs to the actual field devices. At the Station level, the individual Bay IEDs are interconnected to the Station LAN. Local services are usually available at the Station level, such as human-machine interfaces (HMIs) for local parameter loading and configuration of IEDs and diagnostics, and one or more Station level coordinating controllers (IED) for managing protection and control applications between Bays. A wide area network (WAN) gateway (incorporating a firewall) is also usually provided to offer SCADA services to a remote client such as a CC. This overall view lends itself to a structured, hierarchical model to encapsulate the information contents of the Bay and process nodes and the information flows between them. Both horizontal (peer-to-peer) and vertical (client-server) information flows are present, and this is reflected in the mapping of the information model to specific communication stacks. As discussed, the mapping of the IEC 61850 information models to specific communication architectures and protocols can be done in several ways: a typical mapping is to Ethernet, as illustrated in Figure 3.

As may be seen in Figure 3, the communications architecture is partly based upon IP. For vertical communications between Bay IEDs, the Station Controller, HMI, and Gateway, the standard describes the abstract communication services interface (ACSI). ACSI is a unicast, confirmed client-server service model which can be mapped to MMS (International Standards Organization (ISO)/IEC 9506) using TCP/IP to implement the lower MMS stack on the station LAN and process LANs. For horizontal communications between the IEDs, AMUs and IOUs in each Bay and also between Bays in some applications, the standard describes the communication services GSE (generic substation events) and SV (sampled values). GSE may be sub-divided into the sub-types GOOSE (generic object oriented substation events) and GSSE (generic substation state events). GOOSE and GSSE services encode and communicate discrete event-based information such as status updates and commands between IEDs and IOUs. The GSSE service was inherited from the utility communications architecture (UCA), a predecessor of IEC 61850 [11]. Although GSSE is conceptually simpler than GOOSE, it is also less flexible and interoperable and is not recommended for new installations using IEC 61850 [11]. The SV service encodes and communicates sampled data signals such as the sampled analog values generated by AMUs at up to 8 bytes (double precision) resolution. Analog waveforms are typically sampled 60 times per electrical cycle, resulting in a 3 kHz data stream which resembles audio data; the resulting SV data flows must be treated with care and are normally kept within a single Bay and not forwarded outside the relevant process LAN.

All three horizontal communication services are time-critical and use low-level direct mappings into Ethernet frames using non-IP protocols to achieve the required latency for protection applications. They are all multicast message services using an unconfirmed Publish-Subscribe service model with repeated message re-transmission to ensure reliability of delivery. The ASN.1: BER (Abstract Syntax Notation version 1: Basic Encoding Rules) are applied to give lightweight, low-overhead direct encodings of GOOSE and SDV message protocol data units (PDUs) into raw Ethernet frames. To help ensure quality of service (QoS) requirements such as low-latencies are respected, IEEE 802.1Q (priority tagging/VLAN) is also employed for these messages. The GSSE service uses the MMS protocol stack, but replaces TCP/IP at the lower layers with a connectionless open systems interconnect (OSI) stack. GSSE messages use the unconfirmed Information Reporting aspect of MMS and are encoded directly as a string of bits (as opposed to the richer object-oriented dataset carried by GOOSE messages). Wired Ethernet is employed at the physical layer of the process LANs using fiber optic cabling to achieve very high noise immunity. Given the hierarchical structure of the information model and the clear distinction between horizontal and vertical communication services, timing constraints of protection-related traffic can be guaranteed by appropriate selection of VLAN addresses, priorities, and switch forwarding rules. This is such that the GOOSE/GSSE/SMV messages are prioritized over general ACSI (TCP/IP) traffic, and multicast messages are only bridged across LANs when necessary to do so. Since the devices and equipment within each Bay can be considered fixed (or changing very infrequently), VLAN configuration and traffic analysis can be considered a static exercise. As a result, simplified traffic analysis techniques often suffice to bound worst-case response times. A summary of 61850 traffic types and typical timing constraints is given in

Table 1. Main IEC 61850 traffic types and timing constraints. Generic substation events: GSE; generic object oriented substation events: GOOSE; abstract communication services interface: ACSI; sampled values: SV.

Message Type	Application Services	Required Latency (ms)
1A	Fast message (trip): GSE/GOOSE	3–100
1B	Fast message (other): Control, Monitoring	3–100
2	Medium Speed Message: ACSI	100–500
3	Low speed message: ACSI	≥500
4	Periodic raw data: SV	3–100
5	File Transfer: ACSI	≥1000
6	Time Synchronisation: TS	(Required accuracy)

.

The typical mappings of each traffic type in this Table are as given in Figure 3. In addition to the GSE protection-related traffic (type 1A) and SV traffic (type 4), low-latency horizontal traffic may also be comprised of GSE control/automation-related traffic (type 1B). Message latencies for traffic within substations have been studied using network simulators, and are known to be affected by many factors such as the network topology (e.g., ring or star), message configuration (periods, re-transmission rates, payload sizes, and object groupings etc.) and link bandwidth selections [12,13]. Simulation studies and also extensive testing are recommended for a given SAS configuration. For critical messages with the lowest latency requirements (types 1A/1B and 4), it is recommended to keep the publisher and subscriber connected through only a single switch and use fixed message configurations, where possible [13].

2.2. IEC 61850-8-1: Mappings to Manufacturing Message Specification (MMS)

MMS is an international standard (ISO 9506) for the real-time (or near real-time) exchange of supervisory and/or process control information between networked field devices: an introductory overview can be found in [14]. MMS defines a set of standard objects which must exist in a device, upon which primitive operations like read, write, event signaling, and so on can be executed. A major feature of the standard is the description of the virtual manufacturing device (VMD), which every MMS enabled server device must support. The VMD is the main exposed object and all other objects like variables, domains, programs, journals, files etc. are organized under this VMD. The standard describes a set of standard messages which can be exchanged between a client and server station (which hosts the VMD) for the purpose of monitoring and/or controlling these objects through a client application. MMS is a connection-oriented service, and is usually mapped to a standard TCP/IP protocol stack with MMS operating at the application layer. It is essentially agnostic regarding the specifics of an actual application implementation that should be hosted by a VMD, making it applicable across many different domains. The IEC 61850-8-1: Mappings to MMS (ISO/IEC 9506) standard describes how the abstract data models defined in IEC 61850 can be mapped to MMS objects. Although MMS provides a standard and interoperable framework for microgrid management, widely acknowledged and significant drawbacks to its use include a lack of support for publish-subscribe message transference and no ability to negotiate communication channel QoS parameters to obtain real-time guarantees (see [15] and the references therein). Referring to Table 1, this gives some insight as to the motivation for mapping ACSI services to MMS and time-critical services directly to Ethernet.

2.3. The Utility Intranet: Inter-Station Communications

Traditionally, although generation has typically been distributed over several large generating stations operating in parallel with transmission interconnections possibly spanning several countries, these generators and interconnections have been under the control of only a small number of public and private bodies [1,2]. The scheduling, control, synchronization, and management of the overall network have been achieved using well-understood means employing dedicated technologies. With respect to the communication infrastructures, the employed technologies would traditionally consist of dedicated duplex channels between a generation utility and the transmission system operator for telemetry and telecontrol with additional voice/data channels over the public switched telephone network (PSTN) as a backup [3]. Within a smart grid environment, it is generally agreed that Internet Protocol (IP)-based communication networks are most likely to be able to support the connectivity and routing services that are required for smart grid applications [16,17]. It seems likely that the backbone for such an infrastructure will be the “utilities intranet”: a data network which is common to the utilities but separate from the Internet, which is envisioned to allow for the eventual connection of all regional substations, equipment, and CCs throughout the grid [16,17]. Standard best-effort IP links cannot provide the required QoS and network availability requirements to ensure low-latency, low-jitter communications over multiple hops [2,18]. For a supply or generation utility degraded load following results in an inability to meet contractual power requirements, and in the worst-case it may result in the tripping of frequency protection devices and unintentional islanding should spinning reserves be unavailable.

To satisfy the varying range of QoS and routing requirements that smart grid traffic requires [19], differentiated services (DIFFSERV) traffic prioritization and enhanced multicast user datagram protocol (UDP) concepts offered by IP version 6 (IPv6) could be leveraged [16,17,18,19,20]. However, such end-to-end QoS management across multiple control areas is still some way off becoming a practical reality. End-to-end QoS management is needed for distributed control and protection applications. For instance, the stability of the closed-loop while performing bilateral load following (with sampling rates typically in the range 100–1000 ms) has been found to be highly dependent upon the latency (delay), variability in latency (jitter), and packet losses that the control network induces [2,18].

3. IEC 61850 Microgrid Extensions

In the context of microgrids, the IEC 61850-7-420 Communications Standard for DER extends the substation information and communication model to consider distributed resources. This extension to the basic standard is hoped to define the information models and LNs required at the process level for all DER devices, including the electrical connection points, controllers, PV generators, energy converters, DC converters (rectifiers/inverters), and auxiliary systems (measurement/protection devices). IEC 61850-90-7 defines object models of inverters for PV and energy storage systems, and IEC 61850-90-9 considers BESSs. IEC 61400-25 is concerned with the application of the IEC 61850 methodology and object modelling to WTs. With respect to Figure 2 and the IEC 61850 main standards, in microgrid applications each DER interface is considered to be an S-Node (IED) residing at the bay level [7]. The microgrid CC acts as the station controller, and interconnection between the S-Nodes and the main controller is through the station LAN.

A principal difference between the microgrid viewpoint and the substation viewpoint under IEC 61850 lies in the level of distribution of the resources and the degree to which traffic may be isolated between IEDs at the station level. Although much of the required DER interface traffic in a microgrid may be carried using MMS mapped to TCP/IP, support for handling horizontal time-critical GSE and SV traffic—which could be needed for optimized power grid control and protection—could be problematic. Specifically, the need for node discovery and regular system re-configuration in microgrids has been previously identified and methodologies have been developed to allow such behaviors within the IEC 61850/MMS framework [7]. Although one could also attempt to manually configure and re-configure VLANs when nodes come on and off-line within this framework, this could be problematic as it is time-consuming and error-prone. In addition, the possibility of geographical areas spread much wider than a typical substation, and the need for microgrid control traffic to co-exist with other traffic—including regular internet traffic such as email, file transfers and web browsing—is also problematic. For instance, it becomes in effect impossible to follow previous recommendations [13] to ensure that critical IEDs would be interconnected through only a single switch; testing to explore expected message latencies would be difficult to perform on-line.

In this context, we are motivated by aspects of the proposed utility intranet, such as the newly proposed IEC 61850-90-5 Publish-Subscribe Profile which could potentially help to alleviate some of these issues [20]. The 90-5 addendum outlines a proposed methodology whereby GSE and SV traffic may be mapped into multicast UDP/IP frames and DIFFSERV techniques employed within an all-IP infrastructure to bring this class of traffic beyond the sub-station. However, this is very much a new development aimed at wide area networks, and there still remains many outstanding issues (such as the required procedures for setting up and starting/stopping a session) to be solved before it can be relied upon for mainstream use in a large IP-based network or microgrid [20]. In the next Section, however, it is proposed that the integration of IEEE TSN concepts (which are currently implemented as AVB technologies) within an IEC 61850 and MMS/TCP/IP framework provides a flexible and reconfigurable platform capable of overcoming such issues on a smaller microgrid scale (i.e., a single IP subnet or network or LANs bridged at Layer-2).

4. Tunneling IEC 61850 Frames through Current Generation AVB Networks

4.1. Audio Video Bridging (AVB)

AVB is a set of technical standards which have been developed by the IEEE AVB Task Group connected to the IEEE 802.1 standards committee, the goal of which is to provide the specifications that will allow time-synchronized low latency streaming services through IEEE 802 networks [21,22]. The AVB standard defines a node that wishes sends audio/video information as an AVB talker, and a node that wishes to receive audio/video information from an AVB talker as an AVB listener. The AVB standards consist of four principal elements:

IEEE 802.1AS: Timing and Synchronization for Time-Sensitive Applications. When employed with suitably accurate clock sources (e.g., GPS) and supporting network hardware, the 802.1AS protocol enables fault-tolerant clock synchronization over a switched Ethernet network at sub-microsecond levels.

IEEE 802.1QAT: Stream Reservation Protocol (SRP). SRP enables a potential AVB talker to advertise its content (one or more streams) to potential AVB listeners over an AVB-supported network of bridged LANS. SRP also allows the AVB listeners to register to receive the advertised content, and propagate this registration back through the LAN to the AVB talker. If sufficient bandwidth capacity exists on the pathway to guarantee the latency requirements between a potential talker and listener, the stream is set up and VLANs along the route are configured appropriately. On the other hand, if insufficient bandwidth capacity exists, the stream is not set up and both talker and listener are informed.

IEEE 802.1QAV: Forwarding and Queuing for Time-Sensitive Streams (FQTSS). Once AVB streams are confirmed using SRP, FQTSS defines the mechanisms by which the actual time-sensitive traffic is routed and queued. FQTSS specifies a credit based shaper (CBS) for online traffic shaping, and clock synchronization through 802.1AS ensures that all bridges, talkers, and listeners use the same time reference to implement the traffic shapers.

IEEE 802.1BA: Audio Video Bridging Systems. The success of the AVB scheme depends upon the end-to-end support for the above three protocols on the path between any talker and listener (including the network links in between). As such, it is beneficial to identify, for any particular bridge LANs, which devices and pathways could support AVB streaming before attempting to connect any AVB streams. The 802.1BA protocol provides the required procedures to identify participating AVB endpoints and internetworking devices, to identify and flag non-participating devices, and to make this information available in an efficient manner.

Together, these protocols define common QoS procedures and application interfaces for time-sensitive streaming over heterogeneous Layer 2 technologies, effectively defining an API for QoS-related services that extends well beyond the transfer of audio and video [21]. Two classes of service are offered to endpoints by the AVB API: the low-latency “Class A” stream provides guaranteed end-to-end latency of 2 ms (± 125 μs) over seven hops, and the medium latency or “Class B” stream provides guaranteed end-to-end latency of 50 ms (± 125 μs) over seven hops. For applications to take advantage of AVB functionality, AVB transport protocols have been defined in addition to the core AVB standards described above. IEEE 1722 defines the Layer 2 transport protocol for time-sensitive streams using AVB (AVB-TP), and supports transmission of raw audio and video streams encoded using IEC 61883 and other related formats. The 802.1AS clock is sampled before adding the worst-case transport delay (2 ms for Class A streams) to the sample time to get an accurate presentation time to be inserted into the 1722 packets. IEEE 1722.1 is a closely related standard which allows audio video channel discovery, enumeration, connection management, and control (AVDECC) for devices using IEEE 1722 and AVB. The AVDECC protocol allows a remote configuration entity to discover and manage AVB audio/video endpoints and stream advertisement, bandwidth reservation and relinquishment. IEEE 1733 extends AVB functionality to IP Layer 3, allowing real time protocol (RTP) streaming of audio and video over AVB-supported networks, albeit limited to a single IP subnetwork.

4.2. Integrating AVB into an Existing IEC 61850 MMS-TCP/IP Based Infrastructure

It seems clear that with appropriate enhancements, AVB technology using automatic VLAN configuration, bandwidth reservation, and real-time streaming can provide the underlying reconfigurable communication facilities required in a microgrid for transporting horizontal IEC 61850 traffic between DERs. The use of MMS mapped into best-effort TCP/IP traffic could run unaltered within such a scheme. Figure 4 gives an overview of a communications architecture for reconfigurable microgrid command and control which integrates AVB streaming into an existing IEC 61850 MMS-TCP/IP based infrastructure [7]. In the Figure, the S-Node interface separates the ACSI services to run over MMS and TCP/IP in a similar fashion to that Client-Server architecture described in [7]. However, the time-critical services (GSE/SV) are now separated and mapped to AVB streams. Both a stream configuration unit and a message filtering/forwarding unit are virtualized within the MMS framework (see, e.g., [21] for a discussion on such an approach). This is to allow a node, once it and its services have been discovered (using a protocol such as that suggested in [7] for example), to enable the microgrid CC to configure which specific GSE/SV messages are to be mapped to AVB and how the streams are configured, advertised, and connected/disconnected. Although in Figure 4 only one incoming stream from the CC is shown, multiple incoming streams (from both the CC and other DERs) can be realized.

However, although working groups are considering extensions to TSN/AVB functionality into industrial domains such as automotive and process control, standards have not progressed beyond the draft stage and concrete protocols and technology implementations are some way off [21,22]. At the current time, professional products for AVB are almost exclusively restricted to the transport of audio and video. In the next Section, a technique by which these limitations may be overcome is outlined.

4.3. Tunneling IEC 61850 Frames through AVB Streams

A tunneling protocol allows two participating and interconnected nodes to implement a network protocol which is not directly supported by the underlying network which is facilitating their exchange of information. Tunneling can be used to allow a “foreign” or non-supported protocol to run over a network that does not, by default, support that particular protocol. Tunneling involves encoding the raw traffic payload data into a different form by the sender, and decoding into the original form by the receiver(s). Encoding and decoding normally take place at the application layer, and although this usually violates the OSI layering, it is an effective and widely used means to enable a service not normally provided by an underlying network. In the context of the current work, we seek to tunnel the horizontal traffic generated by DERs through streaming audio links. As discussed previously, GSE, SV, and MMS messages are encoded using ASN1.1 rules and the entire application protocol data units (APDUs) are carried in the payload of an Ethernet frame. Therefore, it is highly desired to buffer, frame, and handle the multicast/unicast transmission and reception of a demarked sequence of bytes representing a GSE, SV, or MMS APDU over a current-generation AVB network.

On a current-generation AVB network, there are many possible stream and audio channel configurations, some of which are more efficient in terms of the audio (application) data throughput vs the protocol overheads and stream encoding [23]. The number of channels per stream can be varied, but is upper limited by the AVB-Ethernet packet size (over 60 is possible for encoding 24 bit pulse code modulation (PCM) audio). The number of streams may also be varied, but is also upper limited by the capacity of the AVB switch technology (many hundreds are available in practice on a 1 Gbps network). To enable the tunneling of a sequential block of bytes (a data frame) interspersed by one or more “no data” symbols, a stereo AM824 PCM 48 kHz channel conforming to IEC 61883 is employed. Such a channel can transfer 48 bits of audio data (24 bits on the left channel and 24 bits on the right) per audio sample, at a constant rate of 48,000 samples per s. One or more such channels can be packaged into an AVB stream and transported using the 1722 AVBTP, enabling the transfer of raw audio data between a sender and multiple receivers; moreover, these channels and streams are configurable remotely using the 1722.1 AVDECC protocol. Although intended to carry PCM audio data, the 48 bits per sample can in fact encode any serialized bit or byte stream and can hence be used for tunneling. Given a sequence of bytes to transmit by the sender, each successive block of six data bytes is transferred in successive audio samples; the first three bytes are transferred as the left audio data and the next three as the right. The byte value 00_h is reserved to signal a “no data” symbol, and at least one such symbol is inserted by the transmitter to signal an interframe gap and allow receivers to demark a sequential block of bytes in the serialized byte stream as a frame. Since the byte value 00_h should not then appear as useable data in any frame, some encoding and decoding is required to remove this symbol from appearing in each frame.

To achieve this encoding, the consistent overhead byte stuffing (COBS) algorithm is employed [24]. This procedure allows the encoding of frames to remove the symbol 00_h, allowing its use to be reserved for framing purposes only. Receivers can then decode each received frame to recover the original data. Both encoding and decoding have linear time complexity, and the process of encoding a frame of length x bytes introduces not more than ⌈x/254⌉ stuff bytes to appear in the processed frame. In other words, even in the worst case only one additional stuff byte is inserted for every 254 bytes of user data [22]. This results in a very small loss of useable bandwidth, and the effective bandwidth is reduced to 254/255 of the raw bandwidth. Nevertheless, an effective bandwidth of 48 × 48,000 × 254/255 = 2,294,965 bps is achieved for each stereo channel using this tunneling approach. Since an SRP bandwidth reservation of 8,384,000 bps must be made for such a stream [22], eight streams may be reserved on any single 100 Mbps link (SRP ensures that not more than 75% bandwidth is reserved for AVB) and the overall efficiency of the stream is 27.37%. Although this figure seems low, it is principally due to the combined overheads of Ethernet, IEC 61883, and IEEE 1722, and is comparable to the efficiency of the raw audio bit stream (27.48%) [23]. The effective bit rate of nearly 2.3 Mbps allows a 30-byte frame to be transmitted in 0.1046 ms. Adding the (fixed) 2 ms delay due to the use of synchronized presentation time, plus a worst-case additional latency of 125 μs due to AVB clock synchronization artifacts, the proposed tunneling approach should guarantee that such a message is reliably delivered from the publisher to each subscriber not more than 2.2296 ms after transmission has commenced.

4.4. Prototype Bit-W Device for S-Node Interfacing

In order to manage the streaming and regular data and provide a suitable application layer interface, a prototype “bump-in-the-wire” (Bit-W) device to interface between S-Nodes and a 100 Mbps AVB network switch has been developed. This Bit-W device provides an application interface to send data to a single AVB stream as a talker and also receive data from a single AVB stream as a listener. The device consists of an XS1-L16128QFN multicore microprocessor from XMOS^©, which hosts an IEEE 1722 compliant endpoint and several configurable hardware I2S PCM interfaces, along with a 50 MHz PIC32MX250F128C microcontroller co-processor from Microchip^© which has I2S PCM and USB interfaces. The PIC microcontroller provides the application interface to the S-Node. For streaming data, the device carries out frame buffering using 8 kb transmit and receive FIFOs, performs COBS encoding and decoding, serialization/deserialization of encoded frames to/from a byte stream, and clocking of the bytes to/from the on-chip hardware PCM interface to/from the XMOS device PCM interface. The overall tunneling concept that is proposed is illustrated in Figure 5. Note that although the device could also manage regular IP/Ethernet transmissions with very little additional configuration, this was not considered for our prototype implementation; regular Ethernet traffic is brought directly from the S-Node to the switch. Stream configuration of the Bit-W devices is managed remotely by the microgrid CC by using the IEEE 1722.1 AVDECC protocol. Free-to-use commercial grade software such as UNOS^© Vision^© and AVDECC drivers and libraries are now available to perform such remote discovery and connection control tasks for current generation AVB networks.

Finally, it must be observed that since the rate at which GSE and SV messages could be delivered to the Bit-W device is potentially faster than the rate at which they are sent over the AVB tunnel, there is a risk of small queuing delays in the device due to time waiting in the buffer. Worst-case delays in this situation are, however, relatively simple to analyze using standard techniques [25]. The situation may be further simplified by observing that since delivery of messages tunneled through AVB is extremely reliable due to the use of reserved bandwidth (no loss due to congestion), the stringent exponentially increasing re-transmission requirements for GSE messages in the main IEC 61850 standards (see [11]) can be replaced with pseudo-periodic messaging.

5. Experimental Tests and Validation

5.1. Prototype Testbed and Experiment Configuration

In order to test the proposed technique for tunneling horizontal traffic through a microgrid AVB network, an experimental analysis has been carried out using a prototype testbed. The testbed implementation consists of two S-Nodes connected via Bit-W devices to a small AVB network. The S-Nodes are implemented as 70 MHz LPC2378 ARM-7 embedded microcontrollers from NXP Semiconductors^©. Two 100 Mbps five-port AVB network development switches from DSP4YOU^© form the main communication infrastructure. These OEM switches have an open configuration and are based upon a 400 MHz Mavell^© ARM9 microcontroller pre-loaded with AVB firmware. One S-Node is connected to each AVB switch via a Bit-W device. A trunk cable interconnects the two AVB switches. A photograph of the test bed is shown in Figure 6. Two experiments have been performed using this testbed; the first is concerned with sending GOOSE event data and is described below.

5.2. Experiment One Configuration

An experiment was carried out in order to measure the time taken for a 30-byte GOOSE message to be transferred from one S-Node (publisher) to the other S-Node (subscriber) with varying levels of interference traffic present in the network. Up to 200 Mbps of stochastically generated interference traffic could be presented to the trunk link between the AVB switches; as the capacity of the link is 100 Mbps, levels above this limit generate congestion and potential loss of packets/queuing delays at the switch egress port. The experiment configuration was as shown in Figure 7. Four experiment runs were carried out, and in each run 3600 GOOSE messages were published at a specific level of interference (either 0 Mbps, 50 Mbps, 98 Mbps, or 100+ Mbps). These interference levels represent no interference, half-load on the link, the link at the point of congestion, and beyond the point of congestion. The time taken for the subscriber to receive each message was accurately recorded by the subscriber using an on-chip timer with 0.1 μs resolution in each case. The procedure employed was as follows: the publisher started the timer upon initiation of each message transmission, and monitored a digital input pin for a signal to be asserted; once asserted, the timer was stopped and the value logged. This pin was hardwired to a digital output pin of the subscriber, which asserted the signal upon successful reception and decoding of the message, thus yielding an accurate end-to-end transfer time measurement.

For comparative purposes, each experiment was also repeated once more, in each case sending GOOSE messages through a standard user datagram protocol (UDP) service running over best-effort IP. Each GOOSE message in the UDP configuration was re-transmitted in exponentially increasing intervals in accordance with the main IEC 61850 standard, i.e., after 1 ms, 2 ms, 4 ms, 8 ms, 16 ms, … up to a maximum of 512 ms. In all cases, any message not delivered after 1000 ms are considered to be lost and the test progresses to the next sample.

5.3. Experiment One Results and Discussion

The results obtained for both the AVB tunneled messages and UDP messages transmission times are summarized in

Table 2. Tabulation of experiment one results. User datagram protocol (UDP) and audio video bridging (AVB).

Method	Interference (Mbps)	Mean (ms)	Standard Deviation (ms)	Min (ms)	Max (ms)	>10 ms	>100 ms	>1000 ms
AVB	0	2.0834	0.0000	2.0834	2.0834	0	0	0
	50	2.1130	0.0147	2.0834	2.1251	0	0	0
	98	2.1131	0.0216	2.0834	2.1459	0	0	0
	100+ (Congestion)	2.0803	0.0870	1.8334	2.5417	0	0	0
UDP	0	0.0417	0.0000	0.0417	0.0417	0	0	0
	50	0.0725	0.0379	0.0417	0.1667	0	0	0
	98	0.2537	0.1483	0.0417	1.1251	0	0	0
	100+ (Congestion)	10.7598	54.2260	0.0417	>1000 (513.4414)	321	69	3

. In each experiment, the average (mean) transmission time, standard deviation plus maximum and minimum transmission times which were obtained are recorded. In addition, the number of messages exceeding a transmission time of 10 ms, 100 ms, and 1000 ms (lost messages) is recorded to give an indication of the distribution of the latencies. All measurements include computational overheads due to local message processing times on sending and receiving nodes.

From these data, it can be observed that the end-to-end transmission times for AVB remain consistently around the 2.1 ms level, with very low standard deviation, even in the case of network congestion. No messages were lost or took longer than 2.5417 ms to be delivered and processed in all cases; as mentioned in Section 4.2, the Theoretical maximum transmission time is 2.2296 ms for a 30 byte message. This indicates that the node processing times are of the order 0.3121 ms, which is reasonable considering that it includes several operations including COBS encoding/decoding. An interesting observation in the case of the congested network using AVB was that the recorded minimum transmission time was reduced to 1.8334 ms for a very small number of samples. The reasons for this are not clear at this point, although we suspect it is an artifact of the underlying AVB protocol working close to its operational limits (the theoretical minimum transmission time is 250 μs less than the maximum due to AVB clock synchronization accuracy, yielding a value of 1.9796 ms). Further investigation is warranted in our future work to determine the source of the transient violation in presentation time. A related interesting future area to explore would be the impact of complete removal of the use of presentation times by the AVB listeners, which would further reduce transfer times reported (by up to 2 ms) at the expense of an increase in jitter. These overall results, nevertheless, are indicative that the proposed scheme is suitable for both teleprotection applications (with deadlines of order of 10 ms) and telecontrol application (with deadlines of order of 100 ms).

For the UDP transmission times, however, it can be observed that when the network is not congested, the average and maximum transmission times are significantly lower than that of AVB. No message takes longer than 1.1251 ms to be delivered. The standard deviation, on the other hand, is always higher than that of AVB indicating increased jitter of transmission. Since jitter is in many ways worse than fixed latency in networked control applications [26], AVB provides a preferable solution as variability is reduced at the expense of a small, fixed value of latency. For the case of network congestion, however, the performance of UDP is affected quite dramatically. The mean and standard deviation of transmission time both increase to levels significantly above that of AVB. In addition, we observed that 321 messages now failed to meet a 10 ms teleprotection deadline, 69 failed to meet a 100 ms telecontrol deadline, and three messages were not delivered after 1000 ms and therefore considered lost. The largest recorded transmission time for a received (non-lost) message was > 513 ms. Since congestion may potentially occur at any physical link in an IP network connection at any point in time (e.g., induced by a file transfer performed by an FTP server or TCP socket), this would be unacceptable for real-time protection services as a significant loss of service would occur. In contrast, our proposed AVB scheme would be more tolerant to such congestion as our results demonstrate. The second experiment is concerned with sending SV data for control purposes and is described below.

5.4. Experiment Two Configuration

An experiment was carried out in order to measure the control performance when carrying out hierarchical frequency regulation in an islanded microgrid using real-time hardware-in-the-loop (HIL) simulation. This choice of application was motivated by the knowledge that hierarchical frequency regulation in islanded microgrids is known to be sensitive to communication delays [27]. The importance of HIL simulation-based testing for smart grid applications, including the testing and evaluation of dependable communication strategies has previously been highlighted [28]. For this experiment, 30 byte SV messages encoding frequency measurements were relayed from one S-Node (publisher) to the other S-Node (subscriber) every 100 ms, again with varying levels of interference traffic present in the network. Both S-Nodes were also connected to an additional PC running a simple islanded microgrid model in the “Simulink Real-Time”^© environment. The model represents a simplified (but representative) per-unit emulation of a small islanded electrical network experiencing net load changes, with the principal means of dispatchable generation being a small two-stage thermal generator incorporating a coordinated boiler/turbine control system [29]. The input to the generator/controller is the reference real power output; although a real generator has multiple inputs/outputs and is nonlinear over much of its operating range, the local coordinated control system ensures an offset-free, near-linear input/output relationship by coordinating and trimming the main inputs (turbine valve setting, boiler fuel/air input) [29]. As such the main observable dynamics are that of the turbine itself. In our simulation, we use a representative dynamic model for a small two-stage turbine with re-heater [3]. The subscribing S-Node is interfaced to this generator and executes a secondary digital proportional-integral (PI) controller to regulate frequency following load changes by trimming the real power reference to the generator, a well-known and appropriate control strategy to employ [3,27]. The experiment configuration was as shown in Figure 8. The digital PI controller was tuned to minimize the integral of absolute error (IAE) without having more than 10% undershoot following a net load disturbance. A nominal 2.1 ms network delay was assumed for the purposes of controller tuning. Four experiment runs were carried out, and in each run the IAE was measured over a period of 20 min as a series of net load disturbances occurred once every minute while a specific level of interference (either 0 Mbps, 50 Mbps, 98 Mbps, or 100+ Mbps) was injected through the network. For comparative purposes, the experiments were also repeated by sending each SV message through a UDP service running over best-effort IP and using the same series of net load disturbances and interferences.

5.5. Experiment Two Results and Discussion

As may be expected given the results detailed in Table 2, for interference levels at 0, 50 and 98 Mbps the IAE results for both the AVB and UDP enabled control performance remained at a nominal level of 3.194. For the AVB-enabled controller, the performance remained at this nominal level in the 100+ Mbps (congestion) case; a significant difference was, however, observed in the UDP-enabled controller as the delays have become significantly larger and a large proportion of packets will be lost or experience delays in excess of the sampling time of 100 ms. The regulation performance for both approaches in the congested case was as shown in Figure 9. Visually, one may observe the deterioration present in the UDP-enabled control loop when compared to the AVB-enabled control loop.

The IAE measurement for the AVB-enabled controller over the duration of the experiment was again 3.194, whilst the IAE measurement for the UDP-enabled controller was increased 3.753; in other words the measured performance deteriorated by 17.5% in terms of IAE. One may observe that undershoot is effectively doubled following the same net load change for the UDP-enabled controller, and that the peak deviation was increased by around 10%. Again, since congestion may potentially occur at any physical link in an IP network connection at any point in time (e.g., induced by a file transfer performed by an FTP server or TCP socket), this would be undesirable for real-time frequency regulation as a significant loss of regulatory service would occur. In contrast, our proposed AVB scheme would be more tolerant to such congestion as our results demonstrate. In summary, although more work is needed to further explore the concepts presented here, the results we have presented give a promising validation. As a final note in this Section, we observe that by using an alternate PCM encoding such as AAF and employing more efficient packing of channels into streams, the efficiency of the tunneling procedure we have proposed could potentially be improved to above 90% [21], providing a much more efficient solution for use with current-generation AVB networks.

6. Conclusions

In this paper, it has been argued that some low-level aspects of the usual IEC 61850 mapping to Ethernet are not well suited for the case of microgrids due to their need for regular re-configuration and geographical distribution. It has been proposed that the integration of IEEE TSN concepts (which are currently implemented as AVB technologies) within an IEC 61850 and MMS/TCP/IP framework provides a flexible and reconfigurable platform capable of overcoming such low-level issues. The proposed technique has been experimentally validated using a prototype test platform and Bit-W device, the latter of which has been developed to allow tunneling of IEC 61850 traffic through AVB audio streams. Experimental results demonstrate that low level event and sampled data may be reliably transported within the proposed framework, leading to improved protection and control functions. It has been argued that since AVB streams can be flexibly configured from one or more central locations, and bandwidth reserved for their data ensuring predictability of delivery, this gives a solution which is more reliable than a pure MMS-based solution. In a wider context, our results in the microgrid/AVB context presented here give a good indication that in larger smart grid applications, end-to-end resource reservations using IPv6, DIFFSERV and RSVP should help WAN applications for power system management meet QoS constraints in the utility intranet.