1. Introduction
Establishing real-time communication among a variety of power grid components at different voltage levels has been the focus of the communication research community for a while. Power utility companies have had a successful experience with the supervisory control and data acquisition (SCADA) systems in monitoring and securing the high voltage (HV) transmission part of the grid. Communication technologies have also supported similar functions at the more component-dense distribution part of conventional power grids. However, particularly for the distribution part, smart grids expect much higher support from communication technologies. This is required for upgrading the system smart functions such as self-healing, where a system controller can take actions without human intervention. The challenge is to involve distribution systems with a large number of substations, distributed equipment/sensors, and energy resources, especially in locations where establishing a new communication infrastructure is not feasible.
System grids often promote a low energy consumption and a high penetration of local renewable energy generation in order to support future smart cities with sustainability programs including smart buildings [
1,
2,
3]. This requires a direct interaction of distribution system customers with energy management and conservation entities [
3]. However, the increased penetration of conventional and distributed energy sources, with the dynamic involvement of energy consumers, leads to more complicated distribution systems. As a result, profound changes in grid topology and operational schemes are eventually expected to be imposed [
4]. Therefore, upgrading the distribution system communication network infrastructure to allow large data analysis and real-time operation and control strategies is essential to overcoming the foregoing challenges in establishing sustainable smart cities [
1,
3,
5].
The success of achieving a high level of distribution system automation, such as predictive diagnosis for self-healing applications (i.e., automatic fault detection, localization, and power restoration), heavily relies on the availability and efficiency of the underlying communication infrastructure [
6,
7,
8]. Exchanging all the sensed data and commands between the primary and distribution substations should be done reliably and in real time. Reliability implies communication system availability and its ability to guarantee the delivery of data packets. The real-time requirement is addressed by meeting a certain end-to-end delay. For instance, IEEE 1646 standard [
9] provides guidelines for the time constraints to communicate information between intelligent electronic devices (IEDs) for substation integrated protection, control, and data acquisition. The standard classifies messages according to their urgency. For example, the maximum message delivery time required for system protection ranges from
cycle (for applications within a substation) to 12 ms (for applications external to a substation) [
9].
In light of the above, this work investigates the effectiveness of implementing HWNs by the aid of multipath transmission control protocol (MPTCP) in the communication infrastructure of future smart grid distribution networks. MPTCP uses multiple paths to deliver data simultaneously and reliably over multiple TCP subflows. This work proposes that distribution substations (either secondary or primary) be equipped with HWN-ready monitoring and control units (MCUs) with various integrated sensors. These MCUs are capable to communicate over hybrid wireless communication networks via dual-interface transceivers, supporting two different wireless technologies, and are MPTCP capable. The role of the MCUs is to collect data locally from a variety of sensors/intelligent electronic devices such as phasor measurement units (PMUs), current transformer meters (CTs) and potential transformer meters (PTs). An MCU establishes a single TCP connection (sub-flow) through each interface, forming two simultaneous TCP connections to exchange data among the substations (where different data packets are sent via each sub-flow). The objective behind utilizing MPTCP and hybrid communication technologies is to improve the overall throughput, reduce latency, and increase reliability. Indeed, aggregating the bandwidth of hybrid communication networks (e.g., WiFi and cellular 3G/4G) increases throughput and reduces the overall time needed to deliver a certain amount of data. Furthermore, in case high throughput is not a requirement, reliability is ensured as one interface can act as a redundant link at the lower communication layers, whereas the TCP protocol acknowledgment mechanism guarantees packet delivery at the upper layers.
The contributions of this paper are two-fold. First, we propose an HWN-based communication architecture for smart grid distribution networks that can achieve reliable and fast communication between primary and medium/low-voltage secondary distribution substations. We are interested in a hybrid communication architecture that relies on sending time critical grid information distributed over two different wireless network technologies. One is a contention-free licensed network that is based on a per-use cost structure such as 3G/4G networks. The other is a contention-based network running in a license-free industrial-scientific-medical (ISM) band that can be owned by the utility operator. Second, the performance of the proposed architecture is comprehensively studied by extensive computer simulations in realistic scenarios. The study addresses the performance gain and the effect of essential parameters (such as MPTCP maximum congestion window size, packet loss ratio, data rate, and burst size) on the overall link throughput and data transfer latency. It is worth noting that we do not address the tuning of MPTCP’s internal mechanisms; rather, we evaluate the overall HWN performance under different operational parameters to validate its suitability for distribution substations’ inter-communication requirements.
The rest of the paper is organized as follows.
Section 2 reviews existing research works that are related to reliable communication architectures for distribution systems in smart grids.
Section 3 introduces our system model from the viewpoint of future power distribution systems.
Section 4 presents the proposed communication architecture and provides a brief overview of the MPTCP protocol. The performance of the proposed architecture is investigated in
Section 5. Finally,
Section 6 concludes the paper.
2. Related Works
Currently, many researchers are investigating employing wireless communication in various smart grid applications due to its easy deployment, low cost, and high supported data rates [
10,
11]. In [
8], the author presents the design of sensor/data acquisition nodes that can be equipped with wireless transceivers in the system implementation of a predictive diagnosis scheme for safety-critical applications. Moreover, the authors in [
3] discuss the need for the existence of sensor/control nodes with wireless capabilities that can be integrated into a building management system without disruptive changes to building controls. Indeed, the implementation of smart grid functions (such as predictive diagnosis for safety-critical applications [
8], self-healing [
12] or real-time system state estimation) at the distribution level mandates the exchange of a large data volume within a short period of time. This comes as a direct consequence of the very large number of distribution substations and/or monitored entities. Therefore, the underlying communication network needs to be highly reliable and fast. Due to the broadcast nature of wireless communications, identifying smart grid security vulnerabilities, attacks, and the mitigation of them are under focus of a multitude of research works as in [
13] and the references therein. In addition, the authors in [
14] present a comprehensive discussion and some proposed solutions to implement wirelessly-connected secure sensor/control nodes for a home area network, which represents an essential element in building smart power grids.
Few research works address the proposal of supporting the distribution level of power grids with advanced communication capabilities. Yang et al. [
15] propose a satellite-based communication network to allow the exchange of control information among the 11 kV distribution substations in a single hop fashion. This infrastructure is nominally expensive, although the bandwidth is limited to only 9.6 kbps, which leads to a large latency. The author in [
16] proposes a multihop wireless network with cellular frequency reuse as the communication infrastructure for low voltage distribution networks but without considering the reliability of data packet delivery.
Devidas et al. [
17] study the ability of employing hybrid communication technologies (such as WiFi, Zigbee, cellular, and power line carrier) to carry various data types with different quality-of-service (QoS) requirements in a microgrid. The authors proposed a multi-tier heterogeneous network to direct data packets to the most appropriate communication technology for their QoS requirements. However, each tier runs only one communication technology. Kong [
18] also studies the possibility of improving the QoS of grid data packets delivery. For this, a two-tier communication network combining the IEEE 802.15.4g standard and wired communication is proposed. This architecture, however, does not harness the capabilities of HWNs as it depends on disseminating sensed data wirelessly in the first tier by using only single-interface IEEE 802.15.4g-based devices.
Per packet end-to-end reliability is realized at the transport layer by the TCP protocol. In addition to congestion and flow control mechanisms, TCP also provides end-to-end error-free packet delivery. These TCP schemes are vital for the efficient utilization of bandwidth and prevention of packet loss [
19]. In fact, network congestion causes deviations in phasor measurement units (PMUs) and IEDs’ real-time traffic, whose timeliness is essential in detecting and mitigating faults. Even with path bandwidth reservation, traffic congestion can still occur at times of increasing traffic demand and in transient traffic stress situations [
20,
21].
A proxy-like TCP mechanism is introduced in [
22] as a solution to the TCP scalability concerns given the large number of data sources in a smart grid.
To the best of our knowledge, no other research work in the literature proposes and studies the performance of HWNs that employ MPTCP (as a transport layer protocol) in order to provide timely and reliable data transfer for smart grid distribution level applications.
3. Smart Grid Distribution System
Smart grids aim at realizing efficiency and reliability during different system operation modes. They allow advanced distribution management systems with remote controllability, whereas conventional distribution systems utilize local control algorithms [
23]. Admittedly, data sharing among different elements in distribution systems is vital for smart decision-making. This requires transferring large amounts of data for real-time or non-real-time analysis according to the system operation mode and targeted function. However, the available communication infrastructures for distribution systems are not up to this challenge [
24]. For instance, the SCADA systems of most utility companies do not monitor medium/low-voltage distribution substations.
Figure 1 shows the main block diagrams of local/remote wide-area networks required for upgrading conventional distribution systems towards a smart grid. As illustrated in
Figure 1a, the local monitoring and control units aim at continuous metering, control, and protection of main distribution system components. Several local monitoring and control units share information over a wide area network for remote controllability and implementation of different smart grid functions. The lower layer presents the typical physical components of a distribution system and the upper layer presents the centralized control system. The wide area and local communication networks are supposed to provide complete system integration that satisfies smart operation functions as shown in
Figure 1a.
An example realization of the envisioned distribution system illustrated in
Figure 1a is demonstrated in
Figure 1b. The figure shows a part of an underground conventional 30/11 kV distribution system. The network configuration of conventional systems is typically radial in suburban and rural areas but looped in urban centers with high population density [
1]. Compared with higher voltage level systems, there is a much lower level of network monitoring and automatic control that supports real-time operation. Intelligent electronic devices (IEDs) are located in the primary substations for monitoring, metering, control, and protection of outgoing feeders and primary substation transformers. Distribution substation automation can be enhanced by the deployment of smart meters/sensors that support large data set collection and metering functions.
In this paper, the conventional system is upgraded using monitoring and control units (MCUs) with integrated sensors for enhancing the operation monitoring and control. These MCUs are capable of communicating with one another over a hybrid communication network, which operates using two different wireless technologies. The upgraded distribution system model shown in
Figure 1b is employed as a case study for investigating the advantages and challenges of HWNs in handling large data sets for real-time analysis.
In future smart grids, different levels of message classes (in terms of criticality and priority) should be analyzed and reported, in real-time or on-demand in order to realize different system functions. The required message delivery performance varies between a very high-speed (to develop a protection command) or a low-priority on-demand (for monitoring a quality of service). For example, a protection “Trip” command shall be delivered to other local IEDs within a total time of 4 ms or 5 ms for 50 Hz substations. However, fault recorders and power quality monitors, which are reported on demand, can deliver 64 to 256 samples/power system cycle with low priority within several seconds [
9].
In self-healing applications, the proposed MCUs, via appropriate sensors, should process current/voltage signals, generate time tag synchrophasor measurement, and report in real time over the hybrid communication network via the MPTCP protocol to a central location at the primary substation. The synchrophasor (time tags) reporting rates are at sub-multiples of the nominal 50 Hz system frequency in the range of 10, 25 or 50 frames per second [
25]. Typical systems have overall delay of synchrophasor reporting in the 20 ms to 50 ms range [
26].
4. The Proposed HWN Architecture
In this section, we address the proposed communication architecture and shed light on the involved networks and the MPTCP protocol.
4.1. Inter-Networked Distribution Substations
Consider
N distribution substations connected by feeders to a primary substation as depicted in
Figure 1b. The large number of these substations mandates the usage of affordable but reliable communication technology. Our study focuses on wireless technologies since they are easy to install with cost-effective equipment, especially in old substations, where extending a wired infrastructure is hard. We assume that all the MCUs in the considered distribution substations can communicate to the MCU in the primary substation in a single-hop fashion as presented in
Figure 1b (Only one connection between a distribution substation and the primary substation is shown for easy readability). Each MCU is equipped with a dual-interface wireless transceiver. One interface is a WiFi interface that offers a contention-based access using IEEE 802.11 protocol. The other interface connects to a 3G/4G base station directly. Since distribution substations serve the end user premises, we assume that all the
N distribution substations and the primary substation are covered by a single WiFi network. This mimics a worst case scenario that can happen in densely-populated areas or in case the secondary substation role is similarly performed by a pole-mounted transformer. The assumed scenario also suits the case of the existence of a high capacity backbone network with a WiFi access at the edge. Indeed, this backbone can be represented either by a WiFi mesh network or WiFi-based data concentrators connected through a backbone network of a large capacity. In this case, the secondary substations compete for sending their data to the nearest mesh router or data concentrator, which forwards their data via the WiFi mesh or the concentrators’ backbone network to primary substations. The coverage of the WiFi network overlaps with a 3G/4G cellular network. Each 3G/4G link acts as a dedicated link with no packet contention. The cellular network interfaces for distribution substations are connected to the primary substation via the backbone of the cellular network, which is assumed to be of a high capacity with no limiting congestion.
The MPTCP protocol is used as the transport layer protocol. A brief overview of the MPTCP protocol is given in the next section.
4.2. Multipath Transmission Control Protocol (MPTCP)
MPTCP is one among several multi-homing protocols whose aim is to allow concurrent transfer of data over multiple interfaces. At the data link layer, channel bonding is implemented in IEEE 802.11 [
27] to increase the bandwidth by combining two adjacent channels within a given frequency band. The technique suffers, however, from performance issues when the links have different rate and delay characteristics, i.e., the links belong to different communication technologies. Stream Control Transmission Protocol (SCTP), defined in RFC 4960 [
28], was proposed as a multipath protocol able to run TCP’s congestion control on multiple sub-paths. The early version of SCTP, however, used only one path as a primary sub-connection, while the other interfaces were considered as backups. The newer version, Concurrent Multipath Transfer (CMT-SCTP) [
29], has extended SCTP to utilize all the paths concurrently. Similar to CMT-SCTP, MPTCP is also designed to allow simultaneous use of multiple physical paths (heterogeneous or homogeneous) in the network, but it adds a management sub-layer at the transport layer. It has been shown experimentally in [
30] that the MPTCP outperforms CMT-SCTP in most scenarios. Thus, it is selected as the transport layer protocol in our proposed network architecture.
MPTCP acts as a common transport layer that distributes the traffic to different networks in an HWN. Each network forms a sublayer that carries a single TCP connection (sub-flow). Therefore, the MPTCP layer controls how much data is pushed onto each subflow. Besides ensuring reliable data delivery by the usage of acknowledgments and retransmissions, the MPTCP provides flow control and congestion control functions for subflows similar to the TCP protocol. Flow control mandates that the amount of data to be transmitted should be less than the available space in the receiver buffer. Congestion control guarantees that the transmitted data size cannot exceed a certain limit (congestion window size) to be determined by a congestion control algorithm.
As the MPTCP main goal is to assign more data to the better subflow (i.e., the less congested), the criteria for traffic distribution among the underlying subflows has been investigated [
31,
32]. Several MPTCP implementations available in the literature are designed to resolve the negative impact that highly-congested subflows can cause to other subflows. Linked-increases algorithm (LIA) [
33] has been implemented in some operating systems and is utilized in this work. LIA uses a parameter
to control the aggressiveness of a multipath flow. This congestion control algorithm works as follows.
For each acknowledgment (ACK) received on a subflow
i, increase the subflow congestion window
by
where
is the sum of the congestion window sizes of all subflows in the connection and
is the maximum segment size on subflow
i. The value of
can make a path aggressive, but, to ensure fairness, it is calculated such that it does not cause a subflow to be more aggressive than a competing TCP in a shared bottleneck. The parameter
is computed for each subflow considering the current properties of the flow as
where
is the round trip time on subflow
i.
MPTCP connections take place between the substations through dual-interface MCUs. The management sublayer of MPTCP initializes an end-to-end connection with MP-CAPABLE handshake. The two ends initiate the first subflow through one of the interfaces. The other subflow is established next using the MP-JOIN option signal over the other interface. After initialization, the MPTCP management layer distributes the data segments over the two links by means of a scheduling mechanism, whose objective is to provide load balancing, improve throughput, and achieve fairness.