Assessment of Envelope- and Machine Learning-Based Electrical Fault Type Detection Algorithms for Electrical Distribution Grids

Abstract

This study introduces envelope- and machine learning (ML)-based electrical fault type detection algorithms for electrical distribution grids, advancing beyond traditional logic-based methods. The proposed detection model involves three stages: anomaly area detection, ML-based fault presence detection, and ML-based fault type detection. Initially, an envelope-based detector identifying the anomaly region was improved to handle noisier power grid signals from meters. The second stage acts as a switch, detecting the presence of a fault among four classes: normal, motor, switching, and fault. Finally, if a fault is detected, the third stage identifies specific fault types. This study explored various feature extraction methods and evaluated different ML algorithms to maximize prediction accuracy. The performance of the proposed algorithms is tested in an emulated software–hardware electrical grid testbed using different sample rate meters/relays, such as SEL735, SEL421, SEL734, SEL700GT, and SEL351S near and far from an inverter-based photovoltaic array farm. The performance outcomes demonstrate the proposed model’s robustness and accuracy under realistic conditions.

1. Introduction

In modern electrical grids with distributed energy resources, such as wind turbine generator farms or inverter-based photovoltaic (IB-PV) array farms, protective relays play a critical role as digital devices capable of identifying circuit breaker statuses [1,2], classifying fault types [3,4], identifying fault locations [5,6,7], and conveying such information rapidly through light indicators or digital displays. Various types of relay protection logic are in place to execute essential functions [8], such as presenting applications to clear faults [9], localizing electrical faults within the grid or power lines [10], and identifying faulted electrical phases [11]. The initial step for engineers in assessing fault events involves examining the recorded events from relays. Consequently, they are required to assess the electrical fault state of the relay’s behavior using recorded events to accurately identify the nature of the disturbances, a notably time-intensive task [12,13,14].

As electrical faults typically increase the current magnitude, a simple measure of the surge across each phase is employed to identify faulted phases [15]. In scenarios involving microgrids, whether connected to hydroelectric power sources (characterized by high inertia) or not, overcurrent relays can identify the faulted phases [16]. However, this method falls short in microgrids powered by inverter-based renewable energy sources because it is ineffective in distinguishing the nature of the faulted phases [16]. The most frequent electrical fault in an electrical grid is the phase-to-ground fault. This fault can be recognized by monitoring the voltages, which drop at the phases that are faulted [17]. Considering that during a fault state, the current increases while the voltage decreases, the ratio of voltage to current along power lines could serve as a reliable indicator of fault types. Alternatively, a data-driven strategy leveraging phasor measurement unit data has been proposed, which does not need a previous power system knowledge and can identify the most common types of electrical faults [18]. The impedance technique, which assesses both current and voltage magnitudes, offers another alternative for fault type identification [4]. The phase-to-ground fault apparent impedance is based on the distance elements for mho relays [19,20], and it is represented on a resistance–reactance (R-X) diagram. This R-X plot indicates the impedance variation of the power line over time (apparent impedance) [21]. The protection is engaged when the apparent impedance measurements fall within the predetermined R-X circular plane [7]. The admittance algorithm was developed in [22]; it is a presetting method that requires the impedances of the power line sections to be known for computing the admittance measurements for each phase during the fault states, and the phase and ground boundaries are used to identify the electrical fault types.

Machine learning (ML)-aided fault detection algorithms are being developed to meet the increasing data triangulation and demands in power grid systems. These developments are noteworthy given the insufficiency of traditional logic-based protection systems, as faults directly impact system reliability, safety, and cost. These algorithms have the potential to greatly enhance traditional protection and fault detection. For this reason, an artificial neural network-based optimizer has been developed considering the impact of emergencies on smart grids, such as carbon dioxide emissions, curtailments, etc. [23]. The reliability of power systems has been addressed in the literature by introducing a path-based mixed-integer second-order cone scheme [24] and by combining the roulette wheel selection with the Monte Carlo algorithms [25]. Five ML algorithms were investigated in [26] to identify different types of faults and resistances in a DC microgrid. Additionally, using IEEE 123 bus feeder simulated data, an ML-embedded relay was proposed in [27] to classify three fault conditions. Furthermore, a graph convolutional network was used to improve fault diagnosis performance in [28]. For the islanded microgrid system, a convolutional neural network (CNN)-based protection model was designed to detect possible unsymmetrical and symmetrical faults in an inverter photovoltaic grid [29].

Various signal processing algorithms have also been used to increase the prediction accuracy of ML-based fault detection models. For instance, wavelet transform (WT) was used with decision tree (DT) [30] and support vector machine (SVM) [31] to detect faults in power grid signals. Moreover, statistical distribution and Kullback–Leibler divergence of power grid signals were used in [32] to detect anomalies in the signals. Furthermore, using real-world recorded power grid signals from the Grid Event Signature Library repository [33], a spectral correlation function-based feature extraction method was developed in [34] to extract the characteristic behavior of faults in power grid signals. However, it has not yet been combined and tested with any ML algorithm to improve detection accuracy in a real-world environment. Various conventional feature extraction methods [i.e., fast Fourier transform (FFT), amplitude and phase (AP), autocorrelation function (ACF), power spectral density (PSD), and WT] were investigated in combination with ML algorithms in [35,36]; however, the training dataset was simulated based on a 46 kV substation and tested with real data from the same substation. However, these developed ML-based fault detection models have yet to be tested under real-world conditions because they are generally trained and tested on datasets created based on the same power grid system models or parameters. Thus, the literature still demands more generalized ML-based fault detection algorithms for practical system models.

This article proposes a cascaded fault type detection model comprising three main components: anomaly area detection, ML-based fault presence detection, and ML-based fault type detection. The first part uses an envelope-based detector to identify the anomaly region as created in a previous study [37]. However, we enhance this concept by adapting a new filter design to account for the noisier nature of power grid signals collected from meters compared with simulated signals. The second part functions like a switch that determines whether ML-based fault type detection should proceed. It applies the model to detect the presence of a fault in the examined signal by considering four classes (i.e., normal, motor, switching, and fault). If a “fault” is detected in the signal, the second part of the proposed model activates the third part of the system to identify the fault type, including single-line-to-ground (SLG), line-to-line (LL), double line-to-ground (LLG), and three line-to-ground (3LG).

Moreover, in this study, various feature extraction methods (i.e., raw data, AP, ACF, FFT, PSD, and WT) are investigated to achieve maximum prediction accuracy in the ML models of the proposed cascaded system. Additionally, different ML algorithms [SVM, DT, random forest classifier (RFC), and CNN] are evaluated for these ML-based models. To test the proposed method, the electrical distribution grid testbed is emulated in a software environment using different sample rate intelligent electronic devices (IEDs) (i.e., relays and meters) located near and far from the IB-PV array farm. As mentioned earlier, the ML-based fault detection models in the literature are typically trained and tested on datasets created using the same substation configuration or power grid system model. However, these trained models are not well suited for practical cases. Therefore, the dataset for training the proposed system model is generated by a simulation model based on a 46 kV substation, which is not considered in the emulated testbed environment used as a validation set in this study. Consequently, under realistic conditions, the performance of the proposed algorithms is verified using test data prepared with a software–hardware-emulated electrical grid testbed. This testbed uses various sample rate meters, including Schweitzer Engineering Laboratories’ SEL735, SEL421, SEL734, SEL700GT, and SEL351S, positioned near and far from an IB-PV array farm.

2. Distribution System Testbed

2.1. Circuit and Hardware

The circuit and hardware of the testbed are shown in Figure 1; the diagram represents an electrical grid/substation with metering devices. This testbed is focused on simulating different electrical fault test use case scenarios. The one-line diagram in Figure 1 represents an electrical substation that has a sectionalized bus system [38] with power transformers and feeders connected to an IB-PV array farm as shown in Figure 1a. The meters and relays in the testbed rack (Figure 1b) are indicated with numerical values, and their locations are shown in the circuit (Figure 1a). The red highlighted relays and meters in Figure 1 are used in this research. This circuit is based on a previous publication in [22]. In Figure 1, the relays and meters are selected based on their protection, measurement or control functions, and locations. The devices numbered 1 and 2 (SEL451) are overcurrent protection relays located at $12.47$ kV side of utility A substation feeders; devices 3 and 4 (SEL735) and 5 and 6 (SEL734) are power meters located on fuse feeders to measure customer loads in utility A; device 7 (SEL421) is a distance protection relay located at $34.5$ kV side of the utility A substation; device 8 (SEL351S) is an overcurrent protection relay located at the $12.47$ kV side of utility C substation; and device 9 (SEL700 GT) is a protective relay with a synchrony check control function located in the point of common coupling between the grid side and IB-PV array side.

Utility A has a substation with a distribution grid. Utility A’s substation has two transformers with a primary and secondary voltage of $34.5$ and $12.47$ kV, respectively. In this study, the $12.47$ kV power grid with the load feeders is commonly set as a radial system, but it could also be connected to the IB-PV array farm (utility C). Utility C is an IB-PV array farm, and utility B is the fossil fuel source. In the feeder, loads are configured two SEL735 [39] meters and two SEL734 [40] meters. In one power transformer side, a SEL421 [41] relay is set. The SEL451 [11] relays are set at substation feeders. The SEL351S [42] relay is installed at the IB-PV array farm side, and the SEL700GT [43] relay is set at the grid common point of connection. In the real-time simulator (RTS), relays and meters are connected. The relays and meters are synchronized with a time clock system and connected to a trigger recorded event system.

2.2. Substation and Grid Testbed

This study uses the testbed demonstrated in Figure 2. The testbed has three rack units. The RTS rack has an RTS expansion box (to increase the number of analog/digital signals) with the current and voltage amplifiers. The relay/meter rack has relays and meters that are wired to the RTS. The synchronized clock with displays are wired to IEDs (relays and meters). The communication rack has a time clock and remote terminal unit. The time clock can generate the time signal with or without a global positioning system (GPS) antenna because it has an internal clock. The advanced electrical substation grid testbed was created and led by Dr. Piesciorovsky [44,45,46]. This advanced testbed has been used in different research areas like harmonics metering [44], distributed ledger technology [45], relays and meters commissioning in electrical grid testbeds [46], and electrical fault type detections. This advanced testbed has been used with different distributed energy resources (DERs). In [44], the testbed was used as a substation/grid with a wind farm to assess the measured total harmonic distortions for meters and relays with different sampling frequencies. In [45], the testbed was used as a substation/grid with a customer-owned wind farm using a cyber grid guard to monitor the power quality using distributed ledger technology. In [46], the testbed was used as a substation/grid without DERs to assess methods for commissioning relays, meters, and real-time simulator testbeds for different power system applications. In this study, the testbed was used as a substation grid with an IB-PV array farm, to assess the envelope- and machine learning-based electrical fault type detection algorithms.

Table 1. Main research areas, electrical grid characteristics, references and main deliverables of applications using the advanced electrical substation grid testbed.

Main Research Areas	Electrical Grid Chracteristics	References	Main Deliverables
High harmonics metering	Electrical substation grid with wind farm	[44]	Assessment of total harmonic distortions for meters and relays with different sampling frequency
Distributed ledger technology	Electrical substation grid and customer- owned wind farm with cyber grid guard	[45]	Assessment of monitored power quality using a cyber grid guard with distributed ledger technology
Electrical grid testbed commission	Electrical substation grid feeders without DERs	[46]	Assessment of method for commissioning real-time simulator testbeds for different power system applications
Electrical Fault Type Detection	Electrical substation grid with IB-PV array farm	*	Assessment of envelope- and machine learning–based electrical fault type detection algorithms for an electrical distribution grid

[*] This study.

shows the main research areas, electrical grid characteristics, references, and main deliverables of these studies, using the advanced electrical substation grid testbed.

The layers of the testbed are shown in Figure 3. In this testbed, the data are recorded during the simulations. The data are generated by the RTS, and the relays and meters collect these data. The IEDs (i.e., relays and meters) produce the grid data that are synchronized with the time clock system. The testbed’s architecture (Figure 3) have computers and a time clock system (Figure 3a). The physical layers of the electrical substation grid testbed are shown in Figure 3. Level 1 in the physical layers includes the loads, power lines, breakers, transformers, IB-PV array, feeders, and other grid elements. Level 2 is the protection/metering layer that contains the hardware (relays and meters). Level 3 is the automation layer, which includes the clock displays, and communication devices. Level 4 is defined as the control layer.

2.3. Three-Phase Diagram

The three-phase diagram of the distribution system testbed was created based on the one-line diagram in Figure 1a. This diagram is shown in Figure 4, and it was created using MATLAB/Simulink (version 2015b). The three-phase diagram has three electrical utilities: the substation grid (utility A) of $34.5$/$12.47$ kV, the fossil fuel power plant with a transmission/sub-transmission system (utility B) of 69/$34.5$ kV, and the IB-PV array farm (utility C) of $0.575$/$12.47$ kV connected to the electrical grid (utility A). Utility A has two $34.5$/$12.47$ kV transformers connected in parallel and two feeders. Each power line of $12.47$ kV is controlled by a breaker, and the loads are fed by the power lines. These power loads have 50 and 100 T fuses [47]. The three-line diagram is based on a previous diagram used at the electrical substation grid testbed [22,44,45,46], and the IB-PV array farm (utility C) is based on a modified MATLAB/Simulink model to fit there.

In Figure 4, the substation and grid (utility A) are fed by utility B. Utility A consists of the substation, power lines, and loads. Utility C has an IB-PV array farm. The fault resistances for the use case scenarios are implemented into a MATLAB/Simulink three-phase fault block (Figure 4i). It is a programmable phase-to-phase and phase-to-ground fault breaker system that represents the fault ($0.001$ $\Omega$) and ground ($0.01$ $\Omega$) resistances, snubber resistance (1 × 10⁻⁶ Ω), and infinite snubber capacitance. These values correspond to the default values of the MATLAB/Simulink three-phase fault block. In the electrical faults, the effects of the load current and fault resistance are combined. The fault resistance may be particularly high for ground faults, which represent many of the faults on overhead lines. However, all types of electrical faults are performed in this study. The tests for the electrical faults (3LG, LLG, SLG, and LL) at the power line end of utility A are performed with a fault block, which is presented in Figure 4i. The fault block in Figure 4 is set before running the tests for different electrical fault types and is set to start the electrical fault at 20 s with a fault state duration of $0.5$ s (30 cycles). The relays/meters record their data using the trigger event signal circuit (Figure 4h) that generates the signals for recording the events. In Figure 4, the analog signals at different locations from the IB-PV array and electrical fault site are simulated and collected from meters and relays using no similar sampling frequencies. In the IB-PV array side, the SEL351 (Figure 4a) and SEL700GT relay (Figure 4b,c) record events with 32 samples per cycle. In the electrical substation side, the SEL451 (Figure 4d) and SEL421 (Figure 4e) relays record events with 133 samples per cycle. In the load feeder side, the SEL735 meters (Figure 4f,g) record the events with 128 samples per cycle.

2.4. Time Clock System

In the testbed, the time clock system is an important tool. It has allowed researchers to synchronize the time of the data collected from relays and meters for the electrical fault and disturbance events. Consequently, the recorded events from the IEDs (meters and relays) could be identified easily after running the use case scenario tests, comparing the time of the IED events at the disturbance initiation. The satellite-synchronized network clock is connected to all IEDs [48]. Figure 5 shows the time clock system with hardware. The primary and backup time frames are defined by the GPS antenna and SEL2488 internal clock [48], respectively. In the testbed, the synchronized time for the metering devices is generated by the internal clock of the time clock system. It performs the time synchronization of all relays/meters with the same time-stamps to identify the recorded events for the electrical fault test use case scenarios.

2.5. Recorded-Event System

The recorded-event system is a vital tool in the testbed. The signals are generated from the RTS to the IEDs (i.e., meters and relays). In the RT-LAB project, the event signal circuit is built. Figure 6 shows the output card and the signal circuit. As shown in Figure 6a,b, the recorded event system is required to set the time for triggering the signal circuit in the RT-LAB project. In the testbed, the outputs of the event signal circuit are represented by the digital output card of the RTS, which is wired to the relay/meter’s control inputs as presented in Figure 6. The signals of the recorded event system are generated by the clock switch (Figure 6a). The time of the clock switch is set at same time the disturbance event is initiated for the use case tests. The event signals are synchronized at the same time and sent to the SEL relays and meters. As shown in Figure 6, once the event signal is available, the output card from the RTS (Figure 6c) sends a signal to the meters and relays for recording the events at the same time. In the testbed, the event signal circuit is linked to the metering devices’ inputs that receive the DC voltage signal for recording the events.

In the testbed, the relay and meter settings (event total length, sample rate, and pre-length) are defined. Also, the equation of the event trigger signal is defined with a logic condition for recording the events from meters and relays. These logic equations are set as meter and relay control inputs that are wired to the IEDs. The equations of the event signals are set in the metering devices by using a logic “OR”. The IEDs are set using the AcSELerator Quickset software (version $\mathrm{7.2.2.4}$) in the testbed.

Table 2. Settings for meters and relays.

Devices Number (Type)	Event Characteristics
	Prelength [s (Cycles)]	Total Length [s (Cycles)]	Sampling Frequency (kHz)/Sample Rate (Sample/Cycle)	Event Trigger Equation
1 (SEL351S)	$0.1$ (6)	1 (60)	2 (32)	IN106
2 (SEL700GT)	$0.1$ (6)	3 (180)	$0.24$ (4)	IN304
3 (SEL735)	$0.1$ (6)	5 (300)	8 (128)	IN101
4 (SEL735)	$0.1$ (6)	5 (300)	8 (128)	IN101
5 (SEL451)	$0.1$ (6)	3 (180)	8 (133)	IN106
6 (SEL451SV)	$0.1$ (6)	3 (180)	8 (133)	IN206

shows the meter and relay settings.

3. Envelope Detector-Aided, Machine Learning-Based Fault Type Detection Model

Figure 7 illustrates the proposed fault type detection algorithm. It consists of cascaded structures to identify fault regions, their presence, and types. As shown in Figure 7, measurement data are obtained using relays and meters in the testbed. Initially, the envelope detector-based fault region detection extracts the area of interest from the recorded signals. This process keeps the input sample size constant and removed unnecessary parts for the subsequent algorithms. After applying feature extraction to the clipped signals, the extracted features are input into an ML-based fault detection system. This system categorizes the signals as Normal Condition, Motor, Switch, or Fault. If a fault is detected, the algorithm proceeds to a third step to determine the type of fault. A decision block between the second and third steps determines whether the third step should be run based on the output of the second step. The ML models in both steps are trained for different purposes (the first decides the presence of a fault, whereas the second finds the fault type). This two-step structure was shown in previous studies [35,36] to improve prediction accuracy and decrease prediction time. Finally, in the algorithm’s final step, it identifies the fault types as SLG, LL, LLG, or 3LG.

3.1. Envelope Detector-Based Fault Region Detection Model

The envelope detector is a well-known method used in various fields to capture changes in the amplitude of a signal. This study extracts the amplitude behaviors of recorded data from relays and meters, which are referred to as the signal’s envelope. We then employ a fault region (start/end times) detection mechanism based on user-defined thresholds. Because the algorithm processes signals without prior knowledge of the input, the algorithm’s precision could be adjusted based on the investigated signal and user preferences. The fault region detection using an envelope detector is established in two steps, analytical signal conversion and fault detection as shown in Figure 8. The first part involves obtaining the analytic signal, and the second part involves extracting the region of the fault area. The algorithm described in [37] has yet to be investigated with noisy recorded signals under different sample rates. Therefore, beyond the study in [37], the algorithm in this work is improved by applying a band-pass filter with a new configuration. This improvement is necessary because the analytical conversion is directly affected by the noise and harmonics caused by a power grid system, meters, or relays.

As presented in Figure 8, relays and meters apply the band-pass filter to the recorded data. Assuming that the recorded signal is defined as $\mathbf{s}$, the band-pass filter’s outcomes can be defined as ${\mathbf{s}}_{\mathrm{BPF}}=\mathrm{BPF}\left\{\mathbf{s}\right\}$, where $\mathrm{BPF}\{·\}$ is the band-pass filter operation that removes the noise and harmonics by considering the fundamental frequency of input signal ($f_s=60$ Hz). The operation of the analytical signal extraction is named the Hilbert Transform in Figure 8. It fundamentally removes the negative mirror side of the signal and doubles the positive side’s power of the signal in the frequency domain. Thus, the discrete Fourier transform (DFT) operation is employed to filter the signal to obtain its frequency domain as

$$\mathbf{S}\left[k\right]={\displaystyle \sum _{n=1}^N}{\mathbf{s}}_{\mathrm{BPF}}\left[n\right]exp\left(-j2\pi {\displaystyle \frac{(k-1)(n-1)}{N}}\right),\mathrm{for}k=1,2,\dots ,K,$$

(1)

where N denotes the number of samples of the recorded signal, and K indicates the length of DFT operation that equals the number of samples, $K=N$, in this study. Then, the analytic part of the signal is evaluated by making the negative mirror frequency side of the signal equal to zero:

$${\mathbf{S}}_{\mathrm{A}}\left[k\right]=\left\{\begin{array}{cc}2\mathbf{S}\left[k\right],\hfill & k=2,\dots ,M\hfill \\ \mathbf{S}\left[k\right],\hfill & k=1\hfill \\ 0,\hfill & \mathrm{Otherwise}\hfill \end{array}\right.,$$

(2)

where $M=N/2$ if N is an even number, and $M=\lfloor N/2\rfloor +1$ if N is an odd number. Therefore, the analytical signal of the recorded signal can be obtained with inverse discrete Fourier transform operation as

$${\mathbf{s}}_{\mathrm{A}}\left[n\right]={\displaystyle \sum _{k=1}^K}{\mathbf{S}}_{\mathrm{A}}\left[k\right]exp\left(j2\pi {\displaystyle \frac{(n-1)(k-1)}{K}}\right),\mathrm{for}n=1,2,\dots ,N.$$

(3)

As shown in Figure 8, the complex magnitude operation is applied to obtain the envelope behavior of the analytic signal as ${\mathbf{s}}_{\mathrm{ENV}}=\left|{\mathbf{s}}_{\mathrm{A}}\right|$. Because we do not have prior knowledge of the investigated signal, the fault detection part of the algorithm examines the signal’s envelope to find the fault region’s start/end points. Therefore, the average value of the envelope signal is calculated as $\zeta =(1/N){\sum }_{n=1}^N{\mathbf{s}}_{\mathrm{A}}\left[n\right]$, and then the borders are defined based on the user-defined threshold:

$$\begin{array}{cc}\hfill {\zeta }_{\min }=& \zeta -{\displaystyle \frac{\tau \times \zeta }{100}},\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill {\zeta }_{\max }=& \zeta +{\displaystyle \frac{\tau \times \zeta }{100}},\hfill \end{array}$$

(5)

where $\tau$ denotes the threshold value that can be defined by the user. In this study, the threshold values are defined based on the relays and meters as defined in

Table 3. Threshold configuration for each relay and meter.

Relays or Meters	Type of Signal	Threshold ($\mathit{\tau }$)	Percentage of Change ($\mathit{\upsilon }$)
SEL700GT	Voltage	$10\%$	$15\%$
	Current		$20\%$
SEL351S	Voltage	$10\%$	$15\%$
	Current		$20\%$
SEL451	Voltage	$2\%$	$10\%$
	Current		$40\%$
SEL421SV	Voltage	$5\%$	$10\%$
	Current		$40\%$
SEL735	Voltage	$2\%$	$30\%$
	Current		$40\%$

. After achieving the border values of the envelope signal, the border extraction process is applied to the envelope signal as

$${\mathbf{s}}_{\mathrm{ENV}\text{-}\mathrm{BE}}\left[n\right]=|{\mathbf{s}}_{\mathrm{ENV}}\left[n\right]-{\zeta }_{\max }|+|{\mathbf{s}}_{\mathrm{ENV}}\left[n\right]-{\zeta }_{\min }|.$$

(6)

Thus, the fault diagnosis operation in the algorithm is employed by considering the maximum changes between the extracted border values and the examined signal, which is defined as $\Delta \upsilon =(max\left({\mathbf{s}}_{\mathrm{ENV}\text{-}\mathrm{BE}}\right)/max\left(\mathbf{s}\right))\times 100$. Thus, the phase fault status is defined based on the type of signal, and the percentage of change (Table 3) is defined as

$$\begin{array}{cc}\hfill \mathrm{Voltage}=& \left\{\begin{array}{cc}\mathrm{No}\mathrm{Fault},\hfill & \Delta \upsilon =0\hfill \\ \mathrm{Ground}\mathrm{Error},\hfill & \Delta \upsilon \le {\upsilon }_{voltage}\hfill \\ \mathrm{Fault},\hfill & \Delta \upsilon >{\upsilon }_{voltage}\hfill \end{array}\right.,\hfill \\ \hfill \mathrm{Current}=& \left\{\begin{array}{cc}\mathrm{No}\mathrm{Fault},\hfill & \Delta \upsilon =0\hfill \\ \mathrm{Ground}\mathrm{Error},\hfill & \Delta \upsilon \le {\upsilon }_{current}\hfill \\ \mathrm{Fault},\hfill & \Delta \upsilon >{\upsilon }_{current}\hfill \end{array}\right.,\hfill \end{array}$$

(7)

Consequently, the envelope detector-based fault region detection algorithm obtains the region of anomaly based on the diagnosis results, which are given in (7).

3.2. Machine Learning-Based Fault Presence and Type Detection Models

The envelope detector identifies faults’ regions in the examined signal; however, this study uses ML-based models to detect and identify faults. The extracted area of interest in the algorithm’s first part is fed into the ML-based fault presence and type detection model. This section of the algorithm consists of a feature extraction process and two classification layers as shown in Figure 9. Various techniques are investigated in the feature extraction process to obtain the most-effective algorithm. The extracted features are then fed into the first part of classification to recheck the presence of the fault because the envelope detector may not be able to separate the fault from the effects of the motor and switching operations on the examined signal. Therefore, the first part defines the fault presence and provides classification results directly if no fault is detected in the signal. In the case of a fault, the first classification layer triggers the second classification layer to identify the fault type in the signal. In both classification layers, various ML-based models are also evaluated to find the most effective one.

3.2.1. Feature Extraction Techniques

Because the investigated recorded signal contains various types (i.e., voltage and current) and phases (i.e., A, B, and C), the features of the ML input are established considering all types. Thus, ${\mathbf{z}}_{\ell }$ is one of the recorded signals so that $\ell \in \{1,2,\dots ,6\}$ define the type and phase of the signal as given in

Table 4. Defined numbers of various signal types and phases used in the feature extraction process.

Recorded Signal	Signal Types
${\mathbf{z}}_1$	Voltage of phase A
${\mathbf{z}}_2$	Voltage of phase B
${\mathbf{z}}_3$	Voltage of phase C
${\mathbf{z}}_4$	Current of phase A
${\mathbf{z}}_5$	Current of phase B
${\mathbf{z}}_6$	Current of phase C

.

Notably, the feature extraction methods are expressed by considering that the CNN architecture is used in the ML blocks in the algorithm. For this reason, the feature extraction methods’ outcomes are designed as matrix forms; however, they can easily be converted to vector forms using vector conversion algorithms and applied to other ML algorithms (SVM, DT, and RFC).

Raw Data

In this technique, the input data of the ML model are prepared without any signal processing techniques to observe the achievement or drawbacks of any signal processing techniques. Therefore, the input matrix for the ML block is ready for raw data by combining whole types and phase-in together as

$${\mathbf{X}}_{\mathrm{RAW}}={\left({\mathbf{z}}_1,{\mathbf{z}}_2,\dots ,{\mathbf{z}}_6\right)}^T\in {\mathbb{R}}^{6\times N}.$$

(8)

Fast Fourier Transform

The FFT process is employed to observe the frequency behavior of fundamental signals and harmonics with noise. These signals and harmonics are used as inputs for ML models to find changes based on fault. Thus, the DFT operation is employed to obtain the frequency domain components of the analyzed signal, which is also used in (1) as [49]:

$${\mathbf{Z}}_{\ell }\left[k\right]={\displaystyle \sum _{n=1}^N}{\mathbf{z}}_{\ell }\left[n\right]exp\left(-j2\pi {\displaystyle \frac{(k-1)(n-1)}{N}}\right).$$

(9)

Before the creation of the feature matrix model for FFT, half of ${\mathbf{Z}}_{\ell }$ is removed as ${\mathbf{Z}}_{\ell }^c\left[\alpha \right]={\mathbf{Z}}_{\ell }\left[\alpha \right]$, where $\alpha =1,2,\dots ,\lfloor N/2\rfloor +1$ because the ${\mathbf{Z}}_{\ell }$ includes the same positive and negative sides of the frequency domains. Then, the feature matrix model is created for FFT as

$${\mathbf{X}}_{\mathrm{FFT}}={\left(\Re \left\{{\mathbf{Z}}_1^c\right\},\Im \left\{{\mathbf{Z}}_1^c\right\},\Re \left\{{\mathbf{Z}}_2^c\right\},\Im \left\{{\mathbf{Z}}_2^c\right\},\dots ,\Re \left\{{\mathbf{Z}}_6^c\right\},\Im \left\{{\mathbf{Z}}_6^c\right\}\right)}^T\in {\mathbb{R}}^{12\times (\lfloor N/2\rfloor +1)}.$$

(10)

Amplitude and Phase

The effect of fault can easily be observed in the amplitude and phase values of a signal that behaves differently than usual. Thus, the amplitude and phase information of the signal is used to establish the matrix model as

$${\mathbf{X}}_{\mathrm{AP}}={\left({\mathbf{Z}}_1^A,{\mathbf{Z}}_1^P,{\mathbf{Z}}_2^A,{\mathbf{Z}}_2^P,\dots ,{\mathbf{Z}}_6^A,{\mathbf{Z}}_6^P\right)}^T\in {\mathbb{R}}^{12\times N},$$

(11)

where ${\mathbf{Z}}_{\ell }^A=\left|{\mathbf{Z}}_{\ell }\right|$ shows the amplitude of signal, and ${\mathbf{Z}}_{\ell }^P=arctan\left({\displaystyle \frac{\Im \left\{\mathbf{Z}\right\}}{\Re \left\{\mathbf{Z}\right\}}}\right)$.

Autocorrelation Function

The ACF obtains repeated patterns in a noisy environment. As a result, the ACF can be used for feature extraction methods. Thus, the ACF of the examined signal can be calculated as [50]:

$${\widehat{\mathbf{Z}}}_{\ell }\left[\kappa \right]={\displaystyle \sum _{n=1}^N}{\mathbf{z}}_{\ell }\left[n\right]{\mathbf{z}}_{\ell }[n-\kappa ],\mathrm{for}\kappa =1,2,\dots ,N,N+1,\dots ,2\times N.$$

(12)

Thus, the feature matrix for ACF can be created using both halves of the calculated components as

$${\mathbf{X}}_{\mathrm{ACF}}={\left({\widehat{\mathbf{Z}}}_1[1:N],{\widehat{\mathbf{Z}}}_1[N+1:2\times N],\dots ,{\widehat{\mathbf{Z}}}_6[1:N],{\widehat{\mathbf{Z}}}_6[N+1:2\times N]\right)}^T\in {\mathbb{R}}^{12\times N}.$$

(13)

Power Spectral Density

The signal strength is another feature set that can be obtained with the power of the signal. Thus, by using (9), the feature matrix for PSD can be created [35]:

$${\mathbf{X}}_{\mathrm{PSD}}={\displaystyle \frac{2}{f_{s}N}}{\left(|{\mathbf{Z}}_1^c{|}^2,|{\mathbf{Z}}_2^c{|}^2,\dots ,{\left|{\mathbf{Z}}_6^c\right|}^2\right)}^T\in {\mathbb{R}}^{6\times (\lfloor N/2\rfloor +1)}.$$

(14)

Wavelet Transform

The WT is a common method to investigate power signals at different band filter levels and can provide changing sampling information with a Shannon Entropy calculation at each decomposition level [51]. Thus, the Shannon Entropy of each decomposition level for the recorded power grid signal is obtained to observe information changes as

$${\tilde{\mathbf{Z}}}_{\ell }\left[r\right]={\displaystyle \sum _{n=1}^{N_r}}{\Omega }_{\ell }^r\left[n\right]{log}_2\left({\Omega }_{\ell }^r\left[n\right]\right),\mathrm{for}r=1,2,\dots ,R,$$

(15)

where $N_r$ is the number of samples at the rth decomposition level, and R indicates the number of the decomposition value. In (15), ${\Omega }_{\ell }^r$ shows the probability of energy at the rth decomposition level that can be defined as

$${\Omega }_{\ell }^r\left[n\right]={\displaystyle \frac{|{\phi }_{\ell }^r{\left[n\right]|}^2}{{\sum }_{i=1}^{N_r}\left|{\phi }_{\ell }^r\left[i\right]\right|}}$$

(16)

where ${\phi }_{\ell }^r$ is the high-pass filter outcome in the filter-bank algorithm designed for WT calculation that is defined as [52]:

$${\phi }_{\ell }^r\left[n\right]={\displaystyle \sum _{i=-inf}^{inf}}{\mathbf{z}}_{\ell }\left[i\right]\mathbf{h}[2n-i],\mathrm{for}n=1,2,\dots ,N_r,$$

(17)

where $\mathbf{h}$ is the filter that is designed based on the Daubechies wavelet signal. Notably, the bank filter is established as a cascaded design with high- and low-pass filters. In (17), the first input signal is the investigated signal ${\mathbf{z}}_{\ell }$, but the input signal changes with the low-pass filter’s outcomes when the decomposition number is increased. For instance, the input of (17) is ${\mathbf{z}}_{\ell }$ for $r=1$, and other decomposition values (i.e., $r=2,3,\dots ,R$) are defined with a low-pass filter, which is defined as

$${\vartheta }_{\ell }^r\left[n\right]={\displaystyle \sum _{i=-inf}^{inf}}{\mathbf{z}}_{\ell }\left[i\right]\mathbf{g}[2n-i],\mathrm{for}n=1,2,\dots ,N_r,$$

(18)

where g is the low-pass filter signal. Therefore, the feature matrix for WT can be established using (15):

$${\mathbf{X}}_{\mathrm{WT}}={\left({\tilde{\mathbf{Z}}}_1,{\tilde{\mathbf{Z}}}_2,\dots ,{\tilde{\mathbf{Z}}}_6\right)}^T\in {\mathbb{R}}^{6\times R}.$$

(19)

3.2.2. Machine Learning-Based Classification Layers

Both classification layers in Figure 9 are tested with various ML algorithms. The considered ML algorithms are well-known techniques, including CNN, SVM, DT, and RFC. Each ML algorithm is trained with the created dataset. For the training dataset of the ML models, a simulation model is designed in a MATLAB Simulink environment by taking into account the parameters of the 46 kV substation, which is different from the Figure 2. Basically, we train our model with another substation and apply the trained model to detect the faults in the testbed utilized in this study. The substation model used in the training model has feeders with transformers, capacitor banks, line impedances, and individual loads. As the purpose of ML models is either finding the presence of fault or classifying the type of fault, various fault types are used at the end of each feeder. As we need more different datasets for each specific type of fault, we change the main components of each training datum such as the analog-to-digital converter noise, the load percentage of network, bit number, and start/end sample points of faults. The training process for each ML algorithm used 2113 sample data created using parameters of the real 46 kV substation. However, the training dataset is divided into $70\%$ for training and $30\%$ for validation because the validation data are used to monitor overfitting during the training process. If there is an increasing gap between validation and training performance, the training operation is automatically stopped to avoid overfitting. Because both classification layers are used for different purposes, the training labels and datasets can be different even though they have the same architectural design. The first classification layer is designed to identify a fault in the power signal and separate the fault signal from the switching and motor signal waveforms commonly available in power grids. Therefore, the labels for the first classification layer’s dataset are prepared considering the Normal Condition, Motor, Switching, and Fault. The second classification layer aims to identify the common types of faults, so the labels for this layer are designed as SLG, LL, LLG, and 3LG. Because of the changes in the size of the feature extraction outcomes and label number, the structure of the ML architecture is adjusted, and models are created for each possible variation to achieve the maximum performance in this study.

Convolution Neural Network

The empirically designed CNN architecture for both classification layers is presented in Figure 10. Because the input size can change based on the feature extraction methods, the channel length numbers are only given in Figure 10. The CNN is established with three convolution layers, and the channel lengths of these layers are 16, 32, and 16, respectively. After each convolutional layer, the Leaky ReLU is employed to prevent vanishing problems in the ML layers. As illustrated in Figure 10, the last convolution layer is shaped as flattened layers, and then the flattened layer is connected to two fully connected layers. The lengths of the fully connected layers are empirically adjusted as 64 and 16. The last layer is the softmax layer configured as 4 for both classification layers because the number of labels for each classification layer is the same.

Support Vector Machine

SVM is another ML technique that aims to maximize the space between the defined hyperplane and the nearest data points [31,35]. When dealing with datasets containing more than two labels, optimizing the SVM model can be challenging. For this reason, a polynomial order of four is used to achieve the highest prediction accuracy in this study. Additionally, a one vs. one decision function is employed for the multilabel design.

Decision Tree and Random Forest Classifier

The DT algorithm analyzes the training dataset and labels using a tree-like structure consisting of various nodes (root, decision, and leaf). This structure determines the label for the test data by splitting the dataset into subsets based on the most significant feature. The RFC is an improved version of DT constructed from the collection of DT structures.

4. Testbed Model and Use Case

4.1. Testbed Model

The emulated software–hardware electrical grid testbed is set up with an OP5600 real-time simulator (Opal-RT Corporate, Montréal, QC, Canada)/expansion box with relays/meters in the loop. The real-time simulator is connected to a host computer that is used as an interface to set the test scenarios based on running different temporary electrical faults and selecting the duration of the simulations around 50 s. The emulated software–hardware electrical grid reflects a real-world grid condition because it uses relays and meters in the loop and the simulations are performed with a time step of 50 μs to solve electrical system models and reproduce the nature of electrical events. The testbed model is shown in Figure 11. It is formed by the substation grid that is connected to the IB-PV farm. The meters and relays that are in the red dashed area have different sample rates. In the testbed model, data from the relays and meters are recorded at same time. Therefore, the recorded events from the meters and relays are collected at the same time by using the event signal system.

From the saved relay and meter events, the data (i.e., phase current and voltage signals) are analyzed, and the envelope- and ML-based electrical fault type detection algorithms for the electrical distribution grid with different sample rate relays/meters located at different sites from the IB-PV-array farm are assessed in detail. The test scenarios are based on assessing these algorithms for different temporary fault types at the distribution line of utility A as shown in Figure 11. These temporary electrical faults have a duration of $0.5$ s (30 cycles), and the testbed is prepared for the relays that do not trip the breakers for recording and saving the whole fault states in all protective relays and meters.

4.2. Test Scenarios

The test scenarios for the IB-PV array farm are based on simulating different electrical faults. Four electrical fault tests are performed, and data are collected from six IEDs at the same time. In the IB-PV array farm with the electrical grid, the 3LG, LLG, SLG, and LL electrical faults are simulated. Each fault simulation test is run for approximately 50 s, and the loads are fed by utilities A and C, which are a grid source and IB-PV-array farm, respectively. Then, at 20 s, the temporary electrical fault with a duration of $0.5$ s (30 cycles) is performed at the power line end of utility A. The relays do not trip breakers because the testbed is set for recording and saving the whole fault states in all relays and meters. In these tests, the recorded events from four relays (SEL351S, SEL700GT, SEL451, and SEL421SV) and two meters (SEL735) are collected.

Table 5. Characteristics of IEDs for electrical fault test scenarios.

Devices	Sample Rate (Samples/Cycle)	Electrical Grid Location	Data Point Sites (Figure)
SEL351S	32	IB-PV-array farm	Figure 4a
SEL700GT		IB-PV-array farm	Figure 4b
		Grid side	Figure 4c
SEL451	133	LV side (substation)	Figure 4d
SEL451 SV		HV side (substation)	Figure 4e
SEL735	128	Feeder 1	Figure 4f
SEL735		Feeder 2	Figure 4g

shows the characteristics of relays and meters for the electrical fault test use case scenarios.

The relay and meter events are obtained using the AcSELerator Quickset software (version $\mathrm{7.2.2.4}$) The voltages, currents, and event trigger signals are plotted by the SynchroWAVe software (version $\mathrm{1.9.0.89}$) [53]. The relay and meter events have different sampling frequencies and, consequently, different sample rates. The events from the relays and meters are collected as COMTRADE files and converted into MATLAB files to run the assessment of the proposed envelope- and ML-based electrical fault type detection algorithms.

5. Results

5.1. Relay and Meter Events

The substation grid testbed with the synchronized time system and event signal system for the relays and meters is evaluated satisfactorily by performing the electrical fault test scenarios without tripping breakers with an IB-PV array farm model. The tests are based on a simulation of approximately 50 s (Figure 4). Then, temporary SLG, LL, LLG, and 3LG electrical faults of 30 cycles at the power line end are set at 20 s.

Table 6. Events collected for the electrical fault tests.

IB-PV-Array Farm Test: Use Case Test Scenario Description	Device	Electrical Grid Location	Event ID
Test 1: IB-PV-array and grid with a three line-to-ground fault: Test of approximately 50 s, where the loads are connected to utilities C and A (IB-PV-array and main source). At 20 s, a nonpermanent ABCG fault of $0.5$ s is set at the grid line end (utility A), and the relays do not trip the breakers.	SEL351S	IB-PV-array farm	HR 20535
	SEL700GT	IB-PV-array side	HR 20535
		Grid side
	SEL451	LV-side (substation)	HR 13172
	SEL451 SV	HV-side (substation)	HR 12598
	SEL735	Feeder 1	HR 10141
	SEL735	Feeder 2	HR 10141
Test 2: IB-PV-array and grid with a double line-to-ground fault: Test of approximately 50 s, where the loads are connected to utilities C and A (IB-PV-array and main source). At 20 s, a nonpermanent ABG fault of $0.5$ s is set at the grid line end (utility A), and the relays do not trip the breakers.	SEL351S	IB-PV-array farm	HR 20536
	SEL700GT	IB-PV-array side	HR 10358
		Grid side
	SEL451	LV-side (substation)	HR 13173
	SEL451SV	HV-side (substation)	HR 12509
	SEL735	Feeder 1	HR 10142
	SEL735	Feeder 2	HR 10142
Test 3: IB-PV-array and grid with a line-to-ground fault: Test of approximately 50 s, where the loads are connected to utilities C and A (IB-PV-array and main source). At 20 s, a nonpermanent AG fault of $0.5$ s is set at the grid line end (utility A), and the relays do not trip the breakers.	SEL351S	IB-PV-array farm	HR 20538
	SEL700GT	IB-PV-array side	HR 10360
		Grid side
	SEL451	LV-side (substation)	HR 13175
	SEL451SV	HV-side (substation)	HR 12601
	SEL735	Feeder 1	HR 10144
	SEL735	Feeder 2	HR 10144
Test 4: IB-PV-array and grid with a line-to-line fault: Test of approximately 50 s, where the loads are connected to utilities C and A (IB-PV-array and main source). At 20 s, a nonpermanent AB fault of $0.5$ s is set at the grid line end (utility A), and the relays do not trip the breakers.	SEL351S	IB-PV-array farm	HR 20539
	SEL700GT	IB-PV-array side	HR 10361
		Grid side
	SEL451	LV-side (substation)	HR 13176
	SEL451SV	HV-side (substation)	HR 12602
	SEL735	Feeder 1	HR 10145
	SEL735	Feeder 2	HR 10145

shows the identification events collected from the relays and meters. Figure 12 shows the data from the SEL451 relay for the temporary 3LG electrical fault.

In the use case tests, the recorded event system is triggered at 20 s, and the signal is received by all relays and meters at same time. This result allows the recorded events to be synchronized for all IEDs (meters and relays). These events are collected using the AcSELerator Quickset software (version $\mathrm{7.2.2.4}$). All devices are stamped with the same time for each event test, allowing us to identify and select the relay/meter events for each use case test. The plots for the currents, voltages, and event trigger signal are made by using the SynchroWAVe software (version $\mathrm{1.9.0.89}$) [53], as shown in Figure 12 for the SEL451 relay.

5.2. Envelope-Based Fault Region Detection

Figure 13 shows the performance of the created envelope-based fault region detection algorithm with recorded events from different relays and meters. In the test, we consider various event types (i.e., SLG, LL, LLG, and 3LG) to indicate the proposed method precision. As shown in Figure 13, each type and phase type of the signal are assessed to locate the start/end points of the fault region. Also, the algorithm defines the event based on the user’s defined threshold values. Subsequently, Figure 13a shows an example of the SLG type of signal recorded by the SEL735 (Feeder 2) meter. The result indicates that the proposed method can identify which phase has a fault and its start/end points. As shown in Figure 13b, the test is also applied for the LL type using the SEL351S recorded data. The algorithm still maintains its prediction performance. The algorithm’s performance is also verified for LLG and 3LG fault types, analyzing the recorder signals by SEL451 and SEL735 (Feeder 2), respectively.

5.3. Machine Learning-Based Fault Presence and Type Detection Models

The performance result of the ML-based fault presence detection algorithm is indicated by testing with 20 recorded data from the testbed as given in Figure 14. The various ML-based models and feature extraction methods are used in the test process to find the best option for fault detection. It should be re-emphasized that the training model is trained with the 46 kV substation simulation environment’s dataset, created based on another substation, and has never been used in the testbed. However, the results are shown using a test dataset from the designed testbed.

Further, each combination of ML algorithms and feature extraction techniques is tested with all recorded data from all meters or relays. To clarify, a single model is employed for the entire test dataset; however, we increase the variation of the single model by applying different ML algorithms and feature extraction techniques to identify the best ML-based fault presence and type detection model.

As shown in Figure 14, the prediction accuracy of ML-based fault presence detection is presented in two perspectives: one is ML algorithms, and the other is feature extraction methods. In terms of ML algorithms’ performance, CNN and SVM models provide the maximum prediction accuracy results compared to DT and RFC. Due to the constant tree structure that is not flexible, the prediction accuracy rates of DT and RFC are low for this application. Another reason for this performance decrease on DT and RFC is that the training dataset created using the simulated data of 46 kV substation was never used in the testbed.

In the training process of SVM, the polynomial equation was evaluated to find optimal border between four labels (normal, motor, switching, and fault). The SVM method indicates the best performance prediction accuracy in whole feature extraction methods because each label has unique specific behavior that easy to separate by SVM. The CNN processes patterns and is typically used with image data (2-dimensional data). If the features cannot provide some obvious pattern, the performance of CNN is decreased. As given in Figure 14, the performance of CNN with raw data is $0\%$ because the pattern on the data may break down due to the 2-dimensional conversion of raw data (the vector raw data are split and combined to obtain matrix data suitable for the CNN input). However, the CNN provides promising accuracy with the combination of the amplitude and phase features (AP), frequency domain (FFT), hidden patterns (ACF), and the combination of the time and frequency domains (PSD). The WT is well-known method to find any frequency change in the time domain, which helps to figure out the start/end points of a fault; however, it cannot provide distinct features for fault presence detection application as seen in Figure 14. Accordingly, each ML method cannot achieve significant performance from the WT-based feature extraction technique. Consequently, the CNN and SVM algorithms with the AP, FFT, ACF and PSD techniques provide the best performance for fault presence detection application.

In

Table 7. Different performance metric results of ML-based fault presence detection (classification layer 1) algorithm with various feature extraction methods with ML algorithms.

FE Techniques	ML Algorithms	Precision	Recall	F1-Score
Raw Data	CNN	$0.000000$	$0.000000$	$0.000000$
	SVM	$1.000000$	$1.000000$	$1.000000$
	DT	$0.000000$	$0.000000$	$0.000000$
	RFC	$0.000000$	$0.000000$	$0.000000$
AP	CNN	$1.000000$	$1.000000$	$1.000000$
	SVM	$1.000000$	$1.000000$	$1.000000$
	DT	$0.000000$	$0.000000$	$0.000000$
	RFC	$0.000000$	$0.000000$	$0.000000$
FFT	CNN	$1.000000$	$1.000000$	$1.000000$
	SVM	$1.000000$	$1.000000$	$1.000000$
	DT	$0.000000$	$0.000000$	$0.000000$
	RFC	$0.000000$	$0.000000$	$0.000000$
ACF	CNN	$1.000000$	$0.950000$	$0.974359$
	SVM	$1.000000$	$1.000000$	$1.000000$
	DT	$0.000000$	$0.000000$	$0.000000$
	RFC	$0.000000$	$0.000000$	$0.000000$
PSD	CNN	$1.000000$	$1.000000$	$1.000000$
	SVM	$1.000000$	$1.000000$	$1.000000$
	DT	$1.000000$	$0.350000$	$0.518519$
	RFC	$1.000000$	$0.450000$	$0.620690$
WT	CNN	$0.000000$	$0.000000$	$0.000000$
	SVM	$0.000000$	$0.000000$	$0.000000$
	DT	$1.000000$	$0.250000$	$0.400000$
	RFC	$0.000000$	$0.000000$	$0.000000$

, the classification layer 1 in the proposed model is analyzed with different performance metrics: precision, recall, and F1-Score. Herein, the precision shows how often ML models do correct prediction; recall indicates the frequency of correct label identification; the F1-Score shows the overall performance by considering both the precision and recall performance metrics. Also, each performance metric is indicated between 1 and 0, where 1 is the best and 0 is the worst. As presented in Table 7, each feature extraction technique is investigated with various ML algorithms. The results reveal that DT and RFC algorithms are worst at each feature extraction technique. For instance, each performance metric of DT and RFC is 0 with raw data, AP, FFT, ACF, and WT. On the other hand, the CNN and SVM perform best by employing AP, FFT, ACF, and PSD feature extraction techniques. Except WT and Raw Data, each feature extraction technique provides valuable features for CNN and SVM algorithms to classify labels of data with high precision and recall, but their prediction time varies due to the computational complexity of feature extraction techniques.

The ML-based fault type detection performance measurement is performed by using different relays and meters’ recording data as shown in Figure 15. The predicted outcomes are given in Figure 15 to investigate the response of each ML algorithm and feature extraction method. The 3LG fault type is considered for the test, and the results are indicated with various power relays and meters to investigate the impact of the sample rate and electric grid location of measurement devices. Outcomes reveal that the prediction accuracy is slightly affected by the sample rate measurement devices. For instance, the CNN performance is decreased with the SEL351S, which has 32 sample rates. However, as obtained in the previous results in Figure 14, the SVM shows significant results by making accurate predictions with each feature extraction method except WT. Further, the FFT feature extraction method provides $100\%$ true prediction with the CNN method. The CNN and AP combination can also achieve reasonable results, but other feature extraction methods (i.e., raw data, ACF, PSD, and WT) cannot provide accurate predictions. In the case of other conventional ML algorithms, both DT and RFC only make one correct estimation. In general, the SVM and CNN algorithms perform well in the ML algorithms, and ACF and FFT provide more accurate results than the others.

The ML-based fault type detection (classification layer 2) is also verified with different performance metrics as presented in

Table 8. Different performance metric results of ML-based fault type detection (classification layer 2) algorithm with various feature extraction techniques with ML algorithms. This measurement considers 3LG fault with respect to all power relays and meters.

FE Techniques	ML Algorithms	Precision	Recall	F1-Score
Raw Data	CNN	$0.000000$	$0.000000$	$0.000000$
	SVM	$1.000000$	$1.000000$	$1.000000$
	DT	$1.000000$	$0.200000$	$0.333333$
	RFC	$1.000000$	$1.000000$	$1.000000$
AP	CNN	$1.000000$	$1.000000$	$1.000000$
	SVM	$1.000000$	$1.000000$	$1.000000$
	DT	$0.000000$	$0.000000$	$0.000000$
	RFC	$0.000000$	$0.000000$	$0.000000$
FFT	CNN	$1.000000$	$1.000000$	$1.000000$
	SVM	$1.000000$	$1.000000$	$1.000000$
	DT	$0.000000$	$0.000000$	$0.000000$
	RFC	$0.000000$	$0.000000$	$0.000000$
ACF	CNN	$1.000000$	$0.600000$	$0.750000$
	SVM	$1.000000$	$1.000000$	$1.000000$
	DT	$0.000000$	$0.000000$	$0.000000$
	RFC	$0.000000$	$0.000000$	$0.000000$
PSD	CNN	$1.000000$	$0.400000$	$0.571429$
	SVM	$1.000000$	$1.000000$	$1.000000$
	DT	$1.000000$	$0.400000$	$0.571429$
	RFC	$0.000000$	$0.000000$	$0.000000$
WT	CNN	$0.000000$	$0.000000$	$0.000000$
	SVM	$0.062500$	$0.250000$	$0.100000$
	DT	$0.062500$	$0.250000$	$0.100000$
	RFC	$0.065789$	$0.250000$	$0.104167$

. In this case, all 3LG fault types from testbed are used to measure precision, recall, and F1-Score. Each feature extraction and ML technique is employed. As for pure data (raw data), SVM and RFC provide significant performance; however, the same success of the RFC algorithm cannot be achieved with other feature extraction techniques. Results in Table 8 clearly show that the highest precision and recall can be obtained with the CNN and SVM algorithms along with the AP, FFT, ACF, and PSD techniques. Since ML algorithms are trained on simulated data, their performance is limited by the similar characteristics of labels extracted by using feature extraction techniques. For these reasons, it is necessary to examine not only the prediction accuracy but also the processing times of classification layers 1 and 2.

The processing time of the ML-based fault type identification is presented in Figure 16. The processing time of one datum is measured by considering the specific feature extraction method and ML algorithm. The proposed method is run on the 2023 MacBook Pro (Apple Inc., Cupertino, CA, USA), which has an Apple M2 Max chip, 12-core CPU, 30-core GPU, and 16-core Neural Engine. In Figure 16, the ACF method is excluded due to its extremely high processing time (around $3.5$ s), which ruins the graph visually. Although, the DT cannot provide sufficient prediction accuracy, the lowest processing time is obtained using the DT with FFT method. SVM is the best in the prediction accuracy; however, the computational complexity of SVM fails, with the highest processing time with each feature extraction method. The results reveal that CNN with FFT provides the optimal solution for high prediction accuracy and low processing time in the ML-based fault type identification task of proposed scheme.

6. Discussion

Under real-world conditions, this study tested the fault region presence and type detection algorithms for the emulated electrical substation with an IB-PV array farm. The foremost significant contribution of this study, beyond other studies in the literature, was employing the ML algorithm, which was trained based on the simulation data created with a completely distinct substation model. In the literature, many studies on ML-based fault detection algorithms were tested considering the same substation parameters used for training operations. Please note that data generation is a lengthy process, and it is not always possible to obtain enough data to train a model. However, it was shown in this study that we used a training model from another substation and applied to the distribution system testbed explained in Section 2.

The designed model from the 46 kV substation simulation environment consisted of three main processes: finding the fault region in the data, checking the presence of the fault, and defining the type of fault. The first part of the model was created using an envelope detector-based fault region detection algorithm that was improved from the previous study to achieve higher precision. This part was investigated using data collected from various relays and meters. Because the relays and power-meter hardware components may vary, test data were recorded at different sample rates. However, these variations were effectively avoided in the algorithm’s performance by adjusting threshold values based on the relays and meters in the envelope detector-based fault region detection algorithm.

The second part of the model separated the fault signal from the effect of switching and the motor in the power grid systems. This part was examined using various feature extraction methods and ML algorithms. Because the DT and RFC had tree structures trained based on the other substation, their prediction accuracy was $0\%$ with various feature extraction methods except PSD and WT. However, the $100\%$ estimation performance was achieved with SVM and CNN ML algorithms, but the raw data structure could not provide valuable features for CNN algorithms. For this reason, the ML-based structure in the proposed model was verified with various ML algorithms and feature extraction methods. The third part of the model found specific fault types of the test data. This part was also analyzed using different ML and feature extraction methods. The results revealed that the prediction accuracy for fault type detection was decreasing because the ML models were trained considering different substation parameters. The final results suggest that the specific fault type detection model should be improved to perform better under real-world conditions.

7. Conclusions

This study assessed envelope-aided ML-based electrical fault type detection algorithms for electrical distribution grids with an IB-PV array. The electrical substation grid emulation was established by considering three utilities (utilities A, B, and C). Utility A included a substation ($34.5$ kV) and feeders ($12.47$ kV) fed by utility B. Subsequently, utility B consisted of the generation, transmission, and sub-transmission of a grid. Furthermore, utility C was designed with a substation ($12.47$ kV) and IB-PV array farm. The power grid signals were recorded in the emulated substation using relays and meter equipment. The examination of the proposed algorithms was executed using these recorded power grid signals. The main achievement of this examination was that the ML-based models were trained by considering another substation configuration but were tested with modern electrical grids’ measured power signals. The results provide valuable insights for smart grid systems that leverage the protection and control of power grid systems. For instance, under the test of data from various power relays and meters, the envelope-based fault region detection algorithm obtained an area of interest in the signal (start/end points of fault) precisely. Further, the first ML-based fault presence detection model achieved $100\%$ prediction accuracy and F1-Score by using the SVM algorithm with raw data, the AP, FFT, and PSD features, and the CNN algorithm with AP, FFT, PSD, whereas the second ML-based fault type detection model on specific fault type also reached $100\%$ prediction accuracy and F1-Score with the same ML algorithms and feature extraction method combination. However, the processing time of the algorithms demonstrated the optimal combination for ML models, which was achieved with the SVM algorithm with the PSD feature ($0.057$ s average processing time) and the CNN algorithm with the FFT feature ($0.04$ s average processing time).

Further, as outlined in the IEEE 242 standard [54], the fundamental features of power grid protection system applications specify that the maximum response times for primary and secondary (backup) protection equipment are defined based on the specifications of the relevant power system and fault conditions. In future research, we plan to investigate the applicability of the proposed method in a real-world scenario, taking into account end-to-end time measurements, including fault identification, signal propagation, and protection activation.