*Article* **Application of the First Replica Controller in Korean Power Systems**

#### **Jiyoung Song 1,2, Seungchan Oh 2, Jaegul Lee 1,2, Jeonghoon Shin <sup>2</sup> and Gilsoo Jang 1,\***


Received: 7 May 2020; Accepted: 28 June 2020; Published: 30 June 2020

**Abstract:** The purpose of this paper is to introduce, examine, and evaluate the industrial experiences and effectiveness of a Thyristor Controlled Series Compensator (TCSC) replica controller installed in Korea in 2019 through a review of its configuration, test platform, and practical application, and further to propose operational guidelines for replica controllers. Four representative practical cases were conducted: a Dynamic Performance Test (DPT) under a sufficiently large-scale power system prior to the Site Acceptance Test (SAT), pre-verification for on-site controller modification during operation stage, parameter tuning to mitigate the control interaction, and time domain simulation for Sub-Synchronous Torsional Interaction (SSTI). None of these four cases can be performed in a Factory Acceptance Test (FAT) or on-site. Therefore, TCSC control performance was accurately verified under the entire Korean power system based on a large-scale real-time simulator, which demonstrated its effectiveness as a powerful tool for operations including multiple power electronics devices. Our review herein of these four practical cases is expected to show the usefulness of replica controllers, to demonstrate their strength to deal with practical field events, and to contribute to the further expansion of the application area from a perspective of electric utility.

**Keywords:** replica controller; TCSC; HIL; DPT; real-time simulation; testing; control and protection; large-scale power system

#### **1. Introduction**

With the increasing complexity of power systems and the growth in renewable energy resources, rapidly increasing numbers of power electronic equipment, such as large capacity high-voltage direct current (HVDC) and flexible AC transmission system (FACTS), are being installed to enhance the power system stability and acceptance limit [1–3]. Power systems with multiple, large-capacity HVDC and FACTS are capable of fast, active, and flexible operation, unlike previous passive power systems based on conventional synchronous generators. However, this trend requires a higher level of analysis and operation technology due to the possible severe impacts on power systems and potential unexpected problems such as control interaction [4–6].

The Electromagnetic Transient (EMT) tool is commonly used to analyze power electronics devices, and real-time simulators are used to connect with the actual controller based on EMT [7–9]. Recently, as the area needed to be studied and the number of power electronics devices have increased, the scale of real-time simulators has correspondingly expanded. Typically, Korea, China, Taiwan, and Canada have established environments of large-scale real-time simulators capable of conducting over 1000 buses, and have organized expert departments for analyzing HVDC and FACTS. China Southern Power Grid (CSG) established 33 racks of Real-Time Digital Simulator (RTDS) in 2012, which can contain an entire CSG network above 220 kV. One of the applications of large-scale RTDS is to playback

for several contingencies in real world, such as single-phase to ground fault, main protection faults, and HVDC block. In Operador Nacional do Sistema Elétrico (ONS) in Brazil acquired their 10 racks of RTDS in 2009 to test Rio Madeira HVDC link hardware controller, which is the longest HVDC link in the world. Manitoba Hydro in Canada operates 20 racks of RTDS to study the impact of HVDC links and hydro generation. Southern California Edison (SCE) test control and protection systems, equipment interoperability, and performance under numerous contingency scenarios. Recently, Saudi Electricity Company (SEC) expanded its simulator and now operates more than 40 racks of RTDS. Its purpose is to determine the optimal locations for load shedding for voltage collapse protection, and to study the behavior of the actual controller and the interaction of FACTS with their future HVDC links. Such large-scale real-time expansion is a world-wide trend for various applications, especially for equipment testing under wide area power systems. Additionally, Korea installed 34 racks of RTDS in 2017 to test control and protection systems and to study the interaction of HVDC with FACTS and the network of the entire Korean power system [10,11].

Meanwhile, Hardware In the Loop Simulation (HILS) is to test a performance of an external device and to validate an implementation of a new algorithm or control scheme embedded in actual hardware that is interfaced with a real-time simulator [12]. Moreover, it is mainly applied to one-time tests to approve the pre-performance of an external single device such as protective relay [13,14]. Typically, the device is certified and installed in the field if the performance test is approved in a testing institute or laboratory. However, complicated control and protection systems like HVDC and FACTS should be investigated regularly for continuous analysis of site issues, even after the Site Acceptance Test (SAT). China, France, Canada, and England have established and operated HVDC replica controllers, which have fully replicated the actual on-site controller since early 2010 [15–18]. China has installed all of CSG's HVDC replica controller for study, validation, and maintenance. England established National HVDC Centre to support Caithness–Moray multiterminal HVDC project, which involves multivendors. Therefore, the replica controller was also installed to mainly study interoperability for each converter. Réseau de Transport d'Électricité (RTE) in France owns and installed IFA2000 link replica controller for the purpose of maintenance in 2017 during a refurbishment project. Additionally, in Korea, Korea Electric Power Corporation (KEPCO) will establish replica controller for of all of the HVDC project and FACTS nearby the HVDC converter station.

The replica controller consists of the same hardware board and software as the actual controller. Even though the measurement level can vary depending on the configuration, the internal communication is also the same as that of an actual controller, except for the cooling and redundancy system [16,19]. Therefore, with HILS connecting the replica controller to the real-time simulator, the simulator can reproduce and playback the issues arising in an actual controller. On the contrary, the issues arising in the replica controller may occur in the actual controller. This strength has raised the importance of the replica controller due to its advantage to perform various pre-tests and postanalyses, and to investigate alternatives that cannot be conducted with an actual field controller. In addition, its other applications such as software update of on-site controller, maintenance, and operator training are known to be very wide.

However, the use cases and experiences of the replica controller are not well shared or published. In particular, the procedure of wide area network modeling for the large-scale real-time simulator that is the basis to interface with the replica controller has not been revealed in detail. In this paper, we introduce a replica controller planned by KEPCO, describe its purpose of adaptation, present some practical cases, demonstrate its effectiveness from an electric utility perspective, and suggest further operational directions to operate replica controllers in the future.

#### **2. Replica Controller in the Korean Power System**

#### *2.1. Planning for HVDC and FACTS in the Korean Power System*

Recently, Korea has installed several HVDC and FACTS on its mainland to enhance the power system flexibility and to improve the stability for a 765 kV transmission line fault [20,21]. Since 765 kV line fault significantly impacts on the network, a special control scheme is embedded in HVDC and FACTS to support power system stability during such line fault. For example, Thyristor Controlled Series Compensators (TCSCs) are operated for impedance compensation to reduce 765 kV line loading rate in steady state. In addition, a special control scheme is embedded to boost the impedance compensation level from 50% to 70% for 5 s to improve the transit stability during 765 kV line fault [22].

To maintain a reactive power margin, three static synchronous compensators (STATCOMs) and static var compensator (SVC) are operated in reactive power reserve control mode at both ends of TCSC. In 2025 and 2026, two Line Commutated Converter (LCC) 4GW bi-pole HVDCs will be built nearby the 765 kV line. Figure 1 shows HVDC and FACTS planned for installation in the Korean power system by 2022.

**Figure 1.** HVDC, FACTS in the Korean power system.

A total of 13 HVDC and FACTS will be built in the Korean power system within the next decade, which has increased the need for replica controllers.

#### *2.2. Large-Scale Real-Time Simulator and Wide Area Network Modeling to Interface with the Replica Controller*

KEPCO set up the world's largest RTDS in 2017 to analyze power systems accurately, including multiple HVDC and FACTS at the EMT level [23]. These include 34 racks of RTDS that are capable of accommodating any kind of future HVDC, FACTS, and transmission systems over 154 kV without equivalent network. This large-scale simulator operation requires significant know-how and operational technology. Therefore, KEPCO has developed and operated a preprocessing in-house tool for stable large-scale power system simulations that now can model, stabilize, and verify a large-scale RTDS case of over 1000 buses in about 2 days [24]. Wide area network modeling with real-time simulator (RTS) is very important, since it is the basis for analysis interfaced with replica controller and for study interaction of adjacent other power electronic devices with network dynamics. This paper suggests the use of data conversion functions in RTS in case of large-scale power system modeling of RTS owing to human error of dealing with huge network data. Most of RTS support data conversion function Transient Stability Analysis (TSA) tool to RTS. Procedure of wide area network modeling with RTS interfaced with replica controller is shown in Figure 2.

**Figure 2.** Flow chart of wide area network modeling procedure for real-time simulator (RTS) interfaced with replica controller.

It facilitates detailed analysis of the controller to reveal the dynamic characteristic of a wide area network and the interaction between the facilities and power systems at the EMT level. Figure 3 shows the real-time simulation laboratory established in KEPCO.

**Figure 3.** Real-time Digital Simulator in the Korea Electric Power Corporation (KEPCO) Research Institute.

#### *2.3. Purpose of the Replica Controller*

The main purposes of the replica controller are performing the tests that are unavailable in the field, overcoming the limitations of Software In the Loop Simulation (SILS), and performing more precise and accurate analysis [25–28]. These purposes may vary internationally depending on the configuration of the test environment. KEPCO has defined the following five criteria to set up a replica controller:


For this purpose, KEPCO is planning to set up replica controllers for all HVDC and TCSC in the future, starting with the four TCSC replica controllers installed in 2019.

#### *2.4. TCSC Replica Controller*

The TCSC is installed to compensate for line impedance by the reactance control (boost factor control). The facility configuration consists of a series capacitor, thyristor valve, reactor, and Metal-Oxide Varistor (MOV). TCSC topology and its conceptual control block are shown in Figure 4 [29].

**Figure 4.** Thyristor Controlled Series Compensator (TCSC) topology and conceptual control block.

TCSC topology and the entire network are modeled in a RTS, and the TCSC control and protection systems are implemented on the replica controller. The replica controller receives three analog signals from the real-time simulator: the series capacitor voltage, the thyristor valve current, and the transmission line current. The thyristor firing pulse, the bypass switch, and the boost signal are connected to a digital signal. TCSCs were installed at Shin Youngjoo and Shin Jecheon converter stations. The replica controller is composed of four cubicles: control, protection, operation, and measurement panels. The control, protection, and operation panels have an identical hardware board and embedded software, except for redundancy. Figures 5 and 6 show the actual controller on-site and replica controller in KEPCO laboratory.

**Figure 5.** TCSC actual controller on-site.

**Figure 6.** TCSC replica controller in Korea Electric Power Corporation (KEPCO) lab.

The TCSC replica controller offers easy playback and on-site problem analysis capability, because users can adjust various control settings, protection settings, and several control block parameters on the operator system.

#### **3. Practical Cases**

#### *3.1. Dynamic Performance Test (DPT) with Adjacent Multiple FACTS*

KEPCO recently established a new DPT procedure between Factory Acceptance Test (FAT) and SAT. As the controller DPT is based on large-scale real-time simulator interfaced with a replica controller, it can overcome the barrier of the previous single unit test in FAT. Furthermore, it is a very realistic test because it contains both system dynamics and multiple HVDC and FACTS control characteristics. There are three STATCOM, one SVC, and 765 kV transmission line nearby TCSC. This case covers the verification results of TCSC's boost control scheme to improve the transient stability and the control interaction with nearby parallel FACTS in the event of 765 kV failure, as one of the DPT items.

The 2019 winter operation strategy study data were used as the base case. Large-scale system modeling revealed that RTDS takes 28 racks (based on PB5 processor card), 1399 buses, and 358 generators, and the load is 88,734 MW. There are 22 GW bulk generations over 22 generators on eastern coast of Korea, so two nuclear power plants' trip SPS is applied to maintain transient stability when a 765 kV line fault occurs.

As shown in Figure 7, TCSC maintains a normally capacitive mode (mode: 2) without a bypass or any mode change when a 765 kV line fault occurs, and it follows the boost factor reference appropriately. In addition, both STATCOMs and SVC operate properly to compensate for the low voltage of TCSC and the network near 765 kV transmission line, and to ensure that any control interaction with TCSC does not occur. DPT using the replica controller enables accurate verification of the actual controller's reliability via realistic testing that reflects the network and control characteristics nearby other power electronic devices on-site.

**Figure 7.** TCSC and other FACTS output.

#### *3.2. Preverification of a Modified Function for an Actual Controller*

In the case of TCSC line internal temporary fault, a reinsertion delay of about 1 s was applied in force for stable synchronization of the internal phase lock loop (PLL) controller in the previous control algorithm. After FAT completion, the manufacturer developed a new function called fast reinsertion, and tried to implement it into the actual controller. However, such implementation directly into the actual controller may incur a risk in the absence of any verification using the actual controller. In this practical case, a replica controller is used to pre-verify a new function or a controller improvement to be updated with an on-site actual controller. Figures 8 and 9 show the verification result using a replica controller before and after improvement.

**Figure 8.** TCSC boost factor (before improvement).

**Figure 9.** TCSC boost factor (after improvement).

As shown in Figure 8, TCSC still maintains a bypass mode (mode: 4, 5) for about 1.3 (at 3–4.3 s), even after successful line reclosing. However, after the improvement, TCSC is reinserted immediately after successful line reclosing, as shown in Figure 9. In addition to this practical case, several other tests were conducted to validate the reliability of the new function, and then on-site actual controller was successfully updated.

#### *3.3. Parameter Tuning for Control Interaction*

During commissioning for Shin Jecheon TCSC#1 and #2, at 1 AM (light load) on November 2019, TCSC#2 was permanently bypassed by a sudden reactance error protection operation shown below Figure 10. Analysis of the fault recorder revealed that TCSC #1 and #2 were in a state of oscillation and hunting each other while reaching to the light-load condition at dawn. As a result of modeling the

network condition at that time with a real-time simulator and analyzing it using a replica controller, the error value of the steady-state PLL controller was found to be continuously increasing rather than decreasing.

**Figure 10.** TCSC bypassed on site.

To mitigate this control interaction, the sensitivity of the replica controller was tuned so that the steady-state PLL proportional gain was changed from 4 to 3 and integration gain increased from 400 to 1000 via the heuristic method. Figure 11 shows the conceptual PLL control block.

> ω

θ

**Figure 11.** Phase lock loop (PLL) conceptual PI control block.

The analysis results of the before and after parameter tuning using the replica controller are presented in Figure 12. The oscillation of TCSC was prevented even if #2 had been inserted during TCSC #1 operation, and the steady-state PLL controller error term was also reduced, which verified that the system was under normal control conditions. Several additional tests among the DPT were conducted to verify whether such parameter tuning affects other control performances, and then the on-site controller parameters were finally tuned successfully. Finally, the steady-state PLL gain was suitable for the current power system conditions.

**Figure 12.** Comparison between before/after parameter tuning.

Low-level controller gain tuning does not commonly occur, but it may be needed due to changes in the network condition. Therefore, periodic parameter tuning using the replica controller and the effect on the system should be verified in advance. In June 2020, two 1.4 GW nuclear power plants will be connected to the power system nearby the TCSC sending end, and optimal parameter tuning adapted to the changed network condition will be undertaken again.

#### *3.4. Subsynchronous Torsional Interaction (SSTI) Validation under a Large-Scale Power System*

During the operation stage, the actual SSTI analysis needs to consider the additional damping provided by the adjacent generator, HVDC, and FACTS. Therefore, KEPCO modeled the entire Korean power system without the equivalent network, interfaced with the TCSC replica controller, and applied damping analysis to investigate the possibility of SSTI occurrence. In the event of SSTI, the target generator is modeled with a multi-mass model and time domain simulation for the entire Korean power system is carried out to identify the problem accurately. In this practical case, three STATCOMs and one SVC are considered in addition to the target generator of SSTI analysis. Figures 13 and 14 show the single line diagram and damping analysis result, respectively, for Hanwool nuclear power plant (NP) #2 and TCSC.

**Figure 13.** Single line diagram near TCSC.

**Figure 14.** Damping analysis result.

The Hanwool #3 and #4, which are the target generators for the analysis, are radially connected to TCSC at the N-6 contingency, which has a little negative damping at the generator's minimum output around 12.3 Hz mode frequency. We finally identified that it had positive damping through the time domain simulation for SSTI possibility, as shown in Figure 15.

**Figure 15.** Time domain simulation result in light load of generator.

#### **4. Conclusions**

The importance of replica controllers has recently increased owing to the expansion of HVDC and FACTS facilities. Such replica controllers have been installed mainly in China, Canada, and Europe. In this paper, we described the first replica controller set up in Korea, and presented the more realistic test, analysis, and operation as applicable and necessary under the actual network conditions. In addition, we represented its usefulness for application to the following four practical cases: DPT under a large-scale power system, preverification parameter tuning, event analysis, and SSTI occurrence verification. The replica controllers that demonstrated the usefulness of their on-site application in all four cases could not be performed on-site. The analysis results of these practical experiences offer great value by contributing to the stable future operation of multiple HVDC and FACTS. In particular, since different types and manufacturers of equipment are operated together in the real world, we will prepare and study for interoperability and interaction using multiple replica controllers in the future under an upcoming complicated power system.

Following on from the first TCSC replica controller, KEPCO will continue to setup more HVDC and FACTS replica controllers, and further research and development are being conducted related to its operation and analysis technology for mutual control interaction issues from multivendor replica controllers to adjacent facilities. The integration and comprehensive analysis of multiple replica controllers in the future will require standardization of the replica controllers' technical specifications to optimize the flexibility and interoperability from a perspective of electric utility.

**Author Contributions:** Conceptualization, J.S. (Jiyoung Song) and G.J.; methodology, J.S. (Jiyoung Song) and J.L.; software, J.S. (Jeonghoon Shin). and S.O.; validation, G.L., J.S. (Jeonghoon Shin), and G.J.; investigation, J.S. (Jiyoung Song) and S.O.; resources, S.O. and J.L.; data curation, S.O.; writing—original draft preparation, J.S. (Jiyoung Song); writing—review and editing, J.S. (Jiyoung Song) and G.J.; supervision, G.J.; All authors have read and agreed the published version of the manuscript.

**Funding:** This research received no external funding.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

### *Article* **Remote Laboratory Testing Demonstration**

**Luigi Pellegrino 1,\*, Carlo Sandroni 1, Enea Bionda 1, Daniele Pala 2, Dimitris T. Lagos 3, Nikos Hatziargyriou <sup>3</sup> and Nabil Akroud <sup>4</sup>**


Received: 13 March 2020; Accepted: 28 April 2020; Published: 6 May 2020

**Abstract:** The complexity of a smart grid with a high share of renewable energy resources introduces several issues in testing power equipment and controls. In this context, real-time simulation and Hardware in the Loop (HIL) techniques can tackle these problems that are typical for power system testing. However, implementing a convoluted HIL setup in a single infrastructure can be physically impossible or can increase the time required to test a smart grid application in detail. This paper introduces the Joint Test Facility for Smart Energy Networks with Distributed Energy Resources (JaNDER) that allows users to exchange data in real-time between two or more infrastructures. This tool enables the integration of infrastructures, exploiting the synergies between them, and creating a virtual infrastructure that can perform more experiments using a combination of the resources installed in each infrastructure. In particular, JaNDER can extend a HIL setup. In order to validate this new testing concept, a coordinated voltage controller has been tested in a Controller HIL setup where JaNDER was used to interact with an actual On Load Tap Changer (OLTC) controller located in a remote infrastructure. The results show that the latency introduced by JaNDER is not critical; hence, under certain circumstances, it can be used to expand the real-time testing without affecting the stability of the experiment.

**Keywords:** HIL; CHIL; integrated laboratories; real-time communication platform; power system testing

#### **1. Introduction**

In a global warming trend [1], the key enablers to manage the increasing emission of greenhouse gases (GHGs) are energy efficiency and low-carbon technologies. Renewable sources, storage systems, and flexible loads provide enhanced possibilities. However, these new resources introduce several issues that power system operators have to cope with and investigate. Indeed, an infrastructure with a growing amount of heterogeneous components typically has a higher complexity than traditional power plants [2]. Sophisticated component design methods, intelligent information and communication architectures, automation and control concepts, and proper standards are necessary in order to manage the higher complexity of such intelligent power systems (i.e., Smart Grids) [3–5]. Due to the considerable higher complexity of such cyber-physical systems, it is expected that the validation of smart grid configurations will play a major role in future technology developments.

During the last decade, a growing number of various research and technology development activities have already been carried out in this area. This has been brought to form several research infrastructures (RIs) performing experiments on smart grid activities. However, up to now, no integrated approach for analyzing and evaluating smart grid configurations addressing power system, as well as information, communication, and control topics is available. The integration of cyber security and privacy issues are also not sufficiently addressed by existing solutions. In order to guarantee a sustainable and secure supply of electricity in a smart grid system with considerable higher complexity, and to support the expected forthcoming large-scale roll out of new technologies, a proper integrated RI for smart grids is necessary.

In this context, ERIGrid project [6] came up with a tool for integrating laboratories, exchanging data in real-time. This tool, the Joint Test Facility for Smart Energy Networks with Distributed Energy Resources (JaNDER), allows users to exploit the synergies among the RIs in order to increase the number of test cases that can be implemented without any further cost for hardware/software extension. Obviously, some test cases can be performed in an integrated RI, while others, with strict time constraints, cannot. Indeed, the exchange of information between the RIs introduces a delay due to the communication latency that changes as a function of RI locations, the architecture of the communication infrastructure, and the amount of data to exchange.

There are a few research papers that document real-time simulation using geographically distributed resources. Ravikumar et al. [7] integrated two digital real-time simulator (DRTS) setups based on a customized advanced data acquisition system. Similar setups have been used by Faruque et al. [8] and Rentachintala [9]. In recent works [10] and [11] a geographically distributed real-time simulation (GD-RTS) has been performed using VILLASframework [12], a set of tools that can be used for integrating RIs. In particular, VILLASframework includes a lab-to-lab communication mode, thorough the public internet, with an architecture very similar to that of JaNDER. While VILLAS supports various integration modes with different purposes and performances, JaNDER makes lightness and simplicity of use its major strengths.

In Section 2, a description of the main challenges in power system testing are discussed to explain why HIL techniques and the integration of RIs are important to tackle the upcoming issues in the transition of the power system. Section 3 describes the architecture of JaNDER, and Section 4 presents the results of the characterization tests performed. In the end, Section 5 presents a real application involving two RIs to validate the concept of integrated RIs and to evaluate the performance of JaNDER.

#### **2. Real-Time Simulation Testing**

Real-time simulation and HIL techniques are gaining significant attention as a testing procedure for smart grid applications. These techniques allow the connection of physical devices (e.g., inverters, controllers) to a DRTS that simulates the rest of the power system in real-time. Therefore, instead of using simulated models of hardware equipment, which could be inaccurate compared to the actual device, the actual hardware components are used providing a more realistic view of a smart grid application. Under this setup, exhaustive testing can be achieved in realistic, flexible, controllable, and repeatable conditions leading to fewer problems at the commissioning phase and field operation [13–16].

HIL approaches can be the closest representation that a laboratory test can get to a full hardware application. However, as the HIL setup approaches the actual field implementation, complexity and interfacing challenges increase due to the introduction of various hardware components.

The implementation procedure of such convoluted setup could be quite time consuming. This is derived from the fact that the engineers involved must gain adequate expertise on the different equipment, which could have been developed by various manufacturers, in order to address communication, protection and interface issues between the hardware devices and the DRTS. The physical space required, in the infrastructure hosting the HIL setup, can be another important limitation factor.

To sum up, implementing a convoluted HIL setup in a single infrastructure can be physically impossible or can increase the time required to test a smart grid application in detail. However, platforms like JaNDER, which is described in this work, can be used to overcome the aforementioned issues.

JaNDER allows the interconnection of different infrastructures with very small communication latencies and can be used, as presented in Section 5, to expand the HIL setup. This addresses the space limitation issue and also reduces the time required to interface the different equipment due to the collaboration of different personnel specialized in different components.

#### **3. JaNDER Concept**

To better understand how an HIL setup can be extended on two or more RIs, it is useful to explain how JaNDER works. At the heart of the JaNDER platform, there is the idea that several RIs, linked with a standardized ICT solution, are integrated into a Virtual Research Infrastructure (VRI), encompassing the simulation/experimental potential made available by each laboratory, and thus being able to participate in tests of complex use cases. The overall idea of VRI is depicted in Figure 1.

**Figure 1.** Virtual research infrastructure concept.

Figure 1 shows the advantage of the VRI concept: the possibility for one RI to seamlessly access the resources located at a remote site: these resources might be real field devices, typically mediated through a supervisory control and data acquisition (SCADA) system, or resources modelled inside a DRTS. DRTSs are crucial components of smart grid laboratory experiments but are also very expensive and the possibility of sharing them among RIs is a big advantage. Also, software artifacts such as control algorithms (e.g., a voltage control algorithm, operating on both local and remote devices) can be remotely connected. In practice, measurements and control signals transferred through the VRI's common interface can benefit from higher levels of interoperability—at a basic level, there is the need to transfer plain values associated to devices (either real or simulated), while a more semantic level can be placed "above" the basic exchange of field signals. In fact, this "layered" approach is the one used by the JaNDER platform (under the terminology of "Level 0" and "Level 1"), as described in more details in the following subsections.

#### *3.1. JaNDER Level 0*

JaNDER Level 0 implements the basic mechanism for exchanging live data (i.e., typically measurements and controls) between different RIs. The non-functional constraints associated to this layer are the following:

• Work with a "push" communication model, i.e., do not require any RI to accept incoming TCP connections. This allows for a simpler deployment of the platform.


Given the above requirement, Figure 2 illustrates in a simplified manner the implemented JaNDER Level 0 architecture, which is then explained in details.

**Figure 2.** Joint Test Facility for Smart Energy Networks with Distributed Energy Resources (JaNDER) Level 0 architecture.

The starting point for each RI is a real-time repository used to collect measurements and controls from the field (or more often, as shown in Figure 2, from an already existing SCADA system or DRTS). The reason for this repository is the decoupling of the JaNDER platform from any specific automation solution already installed in the infrastructure. The idea is to have data points from each partner available in the same basic format using a simple key-value repository. The remote connection of remote infrastructures is then implemented by deploying a common real-time repository (which can be hosted in a cloud environment, for example) that is automatically synchronized with all the local real-time databases of the partners. In other words, the common repository acts as a central broker for connecting the different local repositories of the partners, and can be thought as a "virtual bus" connecting all authorized facilities. The real-time repository is an existing open source product, called Redis [17], which is a widely tested, supported, and documented solution with client libraries already developed for many programming languages and environments. However, the HTTPS replication logic is not part of the Redis product and has been developed originally within the project and then released as open-source software.

A crucial implementation aspect of JaNDER Level 0 is the connection between Redis and the existing SCADA system (or individual field devices, depending on each laboratory architecture, such as a DRTS); this link is of course different for each partner, and therefore, the associated software development effort can vary from a really straightforward task, in case the SCADA system provides an open and well-documented interface for connection with external systems to a difficult and time-demanding task, in case there is no central SCADA system and each device must be integrated separately (possibly with different communication protocols). Given the impossibility to address each of the partner's ICT infrastructures in a unified way, it should be noted that the extremely simple data model of the Redis database and the fact that client libraries are available in almost any programming language and environment (including LabVIEW and Matlab, which are highly popular in laboratory environments) helps in keeping the integration effort minimal. In conclusion, all of the requirements mentioned at the beginning of this section have been addressed with the described implementation: the performance argument in particular is detailed in the next section.

The fully open source nature of JaNDER Level 0 [18] makes it easy to extend the VRI community in the future with new participants, however external users will also typically be interested in having a standardized protocol for interfacing: this is handled by the higher JaNDER levels, as described in the next subsection.

#### *3.2. JaNDER Level 1*

JaNDER Level 1 is a software abstraction build on top of Level 0, and its purpose is to provide an IEC 61850 interface on top of the very simple data structures defined in Redis. The reason for adding this level is of course to provide access to the VRI by means of an internationally accepted standard where applicable, such as IEC 61850, which is a standard of primary importance in the field of Smart Grids. The high level architecture of JaNDER Level 1, as an extension of the picture presented for JaNDER level 0, is the following.

In Figure 3, one RI wants to access some contents of its Redis database using the IEC 61850 protocol. It should be stressed that the data points which are in its repository can also come from another RI by means of the JaNDER Level 0 replication mechanism. This means that the IEC 61850 server in the picture can give access to devices in infrastructure 2 but also in infrastructure 1, in a seamless way; at the same time, since the IEC 61850 connection is local to infrastructure 2, there is no need to setup sophisticated cyber security mechanisms for this connection, i.e., the basic MMS protocol without any encryption can be safely used. Of course this is possible because strong cyber security mechanisms are already implemented at Level 0, and are therefore completely transparent for Level 1. The "Mapping" and "CID" files shown as inputs in the above figure are the fundamental inputs needed by the IEC 61850 server in order to work. More in detail, the configured IED description (CID) is the standard IEC 61850 file used for configuring a device (an IED) and contains a data model representing (a subset of) the contents of the Redis repository in terms of IEC 61850 logical nodes. Apart from this file, it is of course necessary to link the data attributes defined inside it with the live values stored in Redis: this is done by means of a mapping file, which is a text file where each line contains an IEC 61850 data attribute name and a corresponding Redis data point name. The server will use this file in order to connect the IEC 61850 data model specified in the CID to Redis.

The IEC 61850 server software is completely open source [19] and based on the OpenIEC61850 Java library provided by Fraunhofer ISE [20] and others in the context of the wider OpenMUC framework. The library has been integrated with a Redis interfacing mechanism which allows the implementation of the behavior discussed above. The interfacing is of course bidirectional, so that measurements can be retrieved and control commands can be issued. (It should be highlighted that the more advanced IEC 61850 control services (enhanced security and select-before-operate) have not been implemented since the direct control with normal security is adequate for the testing scenarios.) In conclusion, the Level 1 of the JaNDER platform provides a simplified way of adding an IEC 61850 standard interface to a RI, built on top of the Level 0 basic solution.

**Figure 3.** JaNDER Level 1 architecture.

#### **4. JaNDER Characterization**

#### *4.1. JaNDER Characterization Introduction*

To better understand the behavior of JaNDER in different contexts a characterization of the different JaNDER levels is necessary. First of all, two of the principal features that are usually taken into account in a distributed system have been considered: latency and response time. In this work only one level has been tested: JaNDER Level 0. The difference between JaNDER Level 0 and Level 1, in terms of latency, is negligible; the delay introduced by JaNDER Level 1 is only few milliseconds due to the mapping between the two layers. For this reason, only a single test has been performed. For this type of architecture, it is important to evaluate the latency to exchange measurements between a single RI and the cloud platform. The next subsection introduces a useful platform to log the measurements used during the tests in order to characterize JaNDER and the test procedure adopted for testing JaNDER Level 0.

#### *4.2. Logging Data Platform: a Big Data Solution*

Since the characterization of JaNDER needs a large number of experiments, to simplify the management of the test, a Big Data solution has been developed. The idea is to automate the analysis of test results as much as possible and to make it easy to manage the collection and analysis of data with a Big Data platform. The platform for data analysis is deployed using the Amazon AWS [21] infrastructure. Databricks [22] was used for the Big Data analysis. Every single RI records the files coming from tests, and at the end of the test, these logs are sent to the single cloud repository (AWS S3 in Figure 4). Using the framework Apache Spark [23] in Databricks, it is possible to aggregate and explore the data coming from all the tests performed by the RI in a single place with powerful Big Data tools and save the results again in AWS S3 to be eventually visualized or processed with other tools.

**Figure 4.** Logging data platform.

To permit that every single test is realized without the overlap of other tests, scheduling the tests is recommended in order to automatically run the test for every RI. In this way, the latency of every single measurement is not compromised by other tests. In the next paragraph, the JaNDER Level 0 test is described.

#### *4.3. Test JaNDER Level 0: Test Description*

The layer considered for this test is JaNDER Level 0. Figure 5 shows the conceptual model of this test. It aims to measure the time required for the data synchronization from an RI to the Cloud Platform: *t*<sup>2</sup> − *t*1. In practice this time should be similar to the latency value of a public network such as the Internet. Since the Internet is not a deterministic network and does not present Quality of Service (QoS), but it is based on best-effort service, obviously the results must always consider this characteristic of the Internet, so it means that sometimes the measured latency may contain outliers with very high values (also in the order of tens of seconds).

**Figure 5.** Conceptual model to test latency for JaNDER Level 0.

Every single RI, at the scheduled time, will send 1000 times a group of measurements. The test is repeated for four times with groups of 1, 10, 100, and 1000 measurements every time. The tests have been executed for a week, in slots of 30 min from 09:00 to 17:00, for each single RI. As already mentioned these experiments were performed using a scheduler in such a way as to have at any moment only a laboratory connected for the tests. The test parameters for JaNDER Level 0 can be summarized as follows:


For this type of test, a lot of data was collected—16 tests a day, four for every partners for four number of measurements. Every test consists of 1000 repetitions, for a total of 16,000 single repetitions for day. This was possible thanks to the automatic script system. Here a test summarizations (after a data cleaning):


Figure 6 shows that during the test the server has no Central Processing Unit (CPU), memory, network problems (of course, there is only one active client). This means that the latency measurement is not influenced by the machine (an Amazon AWS EC2 c3.large).

**Figure 6.** CPU, memory, network measurements during the tests.

#### *4.4. Test JaNDER Level 0: Test Results*

This paragraph shows some results of the JaNDER Level 0 characterization. Considering one of the RI, Figures 7–10 show the latency as function of the number of measurements sent from the RI to the cloud for every single repetitions.

The first set of tests considers one single measurement for repetition. Figure 7 shows the results coming from tests related to a single RI for JaNDER Level 0 test.

In this case the picture on the right shows that there are 3997 measurements exchange between the RI and the cloud platform. The average latency is 32.7 ms, and the third quartile is equal to 29.3 ms.

The second set of experiments considers 10 measurements for time. Figure 8 displays the results. In this case, the third quartile is equal to 31 ms.

**Figure 7.** Single research infrastructure (RI) results for JaNDER Level 0 test: one single measurement for every repetition.

**Figure 8.** Single RI results for JaNDER Level 0 test: 10 measurements for every repetition.

**Figure 9.** Single RI results for JaNDER Level 0 test: 100 measurements for every repetition.

The third experiment considers 100 measurements for time. Figure 9 shows the results. The third quartile is equal to 31.6 ms. The latency is practically in line with previous tests.

The fourth and last test is with 1000 measurements sent from the RI to the cloud platform. Figure 10 shows that the third quartile is equal to 132 ms. This means that with 1000 measurements at a time, the latency increases from about 30 ms to 130 ms.

**Figure 10.** Single RI results for JaNDER Level 0 test: 1000 measurements for every repetition.

Figure 11 presents a summary of the tests. Figure 11 compares all the tests for different number of measurements (1, 10, 100, 1000) and showing the total statistics (grand total). Blue points represent the outlier of every test. The Internet latency, and thus the JaNDER Level latency, is sometime more than one second.

**Figure 11.** RI results for JaNDER Level 0 test.

The JaNDER Level 0 test results confirms that applications with time constant in the range of seconds are feasible with a number of measurements lower than 1000 measurements.

#### **5. Example of Application: Integration of a Remote OLTC Controller**

Real-time testing, using HIL techniques, can evaluate equipment performance in realistic conditions. This could lead in a decrease of their integration time while also reducing potential risks in their field deployment [14]. Components such as controllers and power equipment that are part of actual applications make the HIL setup more realistic and close to the actual conditions.

However, there might be a limitation on how many devices a research infrastructure can host, due to space limitation or due to the fact that different components probably are developed by different manufacturers. Therefore, it might be difficult to investigate if any integration issue occurs between the devices in a safe laboratory environment prior to field implementations. For example, possible issues might exist due to communication delays between the different IED's control devices.

In this section the adequacy of JaNDER platform to serve as the interface between equipment located at remote RIs coupling them in the same HIL setup is investigated.

In order to illustrate that, a Coordinated Voltage Controller, which aims to reduce voltage deviations utilizing different inverters and an On Load Tap Changer (OLTC) located in a Low Voltage (LV) benchmark grid, will be tested in a Control Hardware in the Loop (CHIL) setup. At the same time, it will utilize JaNDER to interact with an actual OLTC controller located in a remote infrastructure.

Finally, in actual field implementations, an interface with an industrial communication protocol is established between a centralized controller and remote IEDs. Therefore, using the JaNDER Level 1 platform, which integrates different components located in remote infrastructures with the industrial protocol IEC 61850, a more realistic environment can be achieved.

#### *5.1. Test Description*

In the aforementioned setup, a centralized voltage controller (CVC) operates in the premises of Electrical Energy Systems Laboratory (EESL) of the National Technical University of Athens (NTUA). The aim of this controller is to minimize the voltage deviations, power losses and the required OLTC operations [14]. In order to do that, the CVC receives measurements such as the load demand, the PVs' production, the batteries state of charge (SoC), and the OLTC's tap position. Then, an optimization problem is solved providing the required setpoints for the PV and battery inverters as well as the OLTC's required tap changes in order to achieve the aforementioned goals. The aim of this optimization is to minimize the voltage deviation from the nominal value while at the same time to ensure that the power losses are not increased considerably, due to the additional reactive power injection, and the OLTC operations are restricted to avoid its wear. This optimization problem is solved with a commercial solver that is able to find a local minimum for non-linear problems. The timestep of each iteration is around 1 min [8].

In this setup, a DRTS located in the EESL, is the backbone of a CHIL setup. Under this setup, the centralized controller can receive and send measurements in real-time. Therefore, both its behavior and its impact on the power system can be investigated under realistic conditions. In the DRTS, the LV Cigre benchmark network is simulated in a similar way to previous works [14,15]. The main goal of the test-case described in this work is to demonstrate the ability of JaNDER to further enhance the setup including an OLTC controller located in a remote installation. Therefore, possible interaction issues between the CVC controller located in EESL in Athens, Greece, and an OLTC controller located in the UDEX laboratory at the Ormazabal premises in Bilbao, Spain, have been investigated. It should be highlighted that the CVC controller has been integrated through the JaNDER Level 1 platform with the OLTC controller. To meet the test setup requirements, an OLTC controller emulator has been used instead of the actual transformer with the OLTC. This controller emulator includes the control of the OLTC and also serves as a real-time simulator of the whole OLTC system. This is achieved by including in the emulator the expected behavior of the OLTC hardware, such as delays in changing taps [24,25]. Therefore, through the JaNDER platform, a HIL setup is interconnected with a custom-made real-time simulator.

JaNDER-Level 1 was used to receive measurements (tap position) and send commands (tap changes) to the OLTC controller. Finally, the tap position measured by the OLTC controller is forced to a simulated OLTC in the DRTS at NTUA's premises in order to include the impact of the actual OLTC's changes to the simulated system. The described advanced setup is presented in Figure 12.

**Figure 12.** Test setup using JaNDER.

#### *5.2. Results*

The operation of CVC controller in parallel with the OLTC controller was tested for a daily scenario with increased PV production. Under this scenario, the CVC is expected to utilize the OLTC in order to mitigate voltage rise issues.

In Figures 13 and 14, the voltage profiles with and without the CVC operation are presented. In Figure 15, the Tap position forced to the OLTC controller by the CVC controller is presented.

**Figure 13.** Voltage profile at consumer premises without the centralized voltage controller (CVC).

**Figure 14.** Voltage profile at consumer premises with the CVC.

**Figure 15.** On Load Tap Changer (OLTC) position changes.

Under this setup, the ability of JaNDER platform to enhance real-time testing was investigated. The CVC was tested and validated against an OLTC controller and real-time simulator designed by specialized personnel in a different location. This test case clearly showed that JaNDER can be used for testing applications, like the CVC controller with equipment in different RIs without affecting its operation due to the time delays introduced by the interface. In Figure 16, the delays measured during a request for a tap position by the CVC to the OLTC are presented. The most significant delay is introduced by the OLTC controller/OLTC real-time simulator which emulates the actual performance of an OLTC in a transformer. The inverters, which are simulated in the DRTS located in the same infrastructure as the CVC controller, receive their setpoint faster, which results in faster acknowledgement of the setpoint implementation by the CVC controller. Therefore, both the distributed DERs implement their setpoints in different time frames and the CVC controller acknowledges those setpoints at different time stamps.

**Figure 16.** Time delays introduced in the setup.

However, the CVC time step (~1 min) is significantly larger than the time required from the CVC controller to send the setpoints to the DERs and OLTC and acknowledge their implementation (~5 s). Therefore, the overall behavior of this application is not affected by the different time frames that the devices react to the CVC controller setpoints.

As shown in Figure 16, the most significant delay is introduced by the OLTC operation according to the CVC request. This delay could differ according to the OLTC manufacturer. It is clear of course that if the OLTC controller's response is comparable to the centralized controller's timestep, the overall behavior of the system would be affected. For example, during online operation the centralized controller initiates a sequence by sending setpoints to all the local devices. If the OLTC controller response is comparable to the centralized controller timestep, then it is possible that the tap position will remain the same when the central controller begins measurements in the next timestep. The central controller would then send an additional request (e.g., tap increase) to the OLTC controller. If the OLTC controller implements those request in series, the resulting tap position would be wrong leading to undesired operation of the OLTC. To avoid this, the CVC controller should request, for example acknowledgement from the local controllers that the ordered setpoints have been implemented. The setup presented can effectively reveal such weaknesses of the controller design in a safe laboratory environment.

In this implementation, the most significant factor of failure is the OLTC response compared to the CVC timestep, since the platform's delays are insignificant compared to the CVC timestep. Nevertheless, the platform delays can affect the overall stability of the setup and the accuracy of the results, if applications requiring faster response are considered. The stability of a control system considering the sample rate of the controller for systems with communication network delays has been investigated in [26,27]. A controller with slower response or slower sample rate is usually a way to increase the robustness of the control against communication delays. This is not applicable in setups with several controllers located in remote locations, since altering their response might lead to a behaviour different than anticipated in the field deployment. A general rule of thumb when using the JaNDER platform to create a testing setup is to compare the network delays introduced by JaNDER, with the actual network delays expected in the field deployment or to ensure that the controller has a sample rate that is not affected by the communication delays. Otherwise, the testing setup is not suitable, as shown in the example of decentralized secondary voltage and frequency control of an islanded microgrid under varying time delays [27].

#### **6. Discussion**

From the point of view of the CVC users, this test case using JaNDER was more realistic by interfacing their DRTS with the industrial product. At the same time, the collaboration with the experienced industrial personnel has assured a safe and fast interfacing of both devices, since every team was able to prepare the equipment in their area of expertise, thus boosting the overall implementation time.

From the point of view of the industrial partner, the JaNDER implementation has provided an insight on state-of-the-art approaches on smart grid applications. At the same time, the introduction of their equipment in such application presented a new opportunity for future research and investigation area.

Since most of commercial real-time simulators have a considerable capital cost, from the overall project perspective, JaNDER can promote collaboration between different infrastructures in order to expand their real-time testing capabilities without spending significant resources in the upgrade of their equipment and can provide considerable benefits for both a commercial and academic field by connecting geographically distributed real-time simulators.

On the other hand, since the industrial protocol (IEC 61850) is widely supported by JaNDER platform, studying the impact of this communication alongside the electrical grid in real-time, can provide a valuable insight on the impact of the delays and interfacing issues of both networks.

Finally, connecting power components (e.g., inverters) to different infrastructures can also be utilized through this platform to study the latency impact and to test slow varying phenomena of these components. Knowing that this platform was proven as a reliable and valuable tool with no critical latency, it is interesting to expand it in real-time testing without affecting the stability of the experiment.

#### **7. Conclusions**

The integration of distributed renewable energy resources to the grid is increasing its complexity and, therefore, new flexible system test methods are needed to study their impact and to ensure their efficiency and security. A novel HIL setup, based on remote hardware integration, is presented in this paper. This new testing setup is available thanks to the real-time communication tool developed in the context of ERIGrid project called JaNDER.

To demonstrate this new testing setup, a real test case, interfacing an industrial device like an OLTC controller with a remote DRTS with a LV network model, has been implemented. The test results shows that the latency for data exchanging due to JaNDER is lower than 300 ms. Hence, this platform is suitable for testing remote hardware/software and analyzing slow varying phenomena. Indeed the latency does not affect the stability of the experiments.

**Author Contributions:** Conceptualization, C.S. and D.P.; Data curation, E.B. and D.T.L.; Project administration, L.P.; Resources, D.T.L., N.H. and N.A.; Software, E.B. and D.P.; Supervision, L.P.; Writing—original draft, L.P., E.B., D.P., D.T.L. and N.A.; Writing—review and editing, L.P., C.S., D.T.L. and N.H. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work is supported by the European Community's Horizon 2020 Program (H2020/2014–2020) under project "ERIGrid" (Grant Agreement No. 654113). Further information is available at the corresponding website www.erigrid.eu.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
