Fast Minimum Error Entropy for Linear Regression

Abstract

The minimum error entropy (MEE) criterion finds extensive utility across diverse applications, particularly in contexts characterized by non-Gaussian noise. However, its computational demands are notable, and are primarily attributable to the double summation operation involved in calculating the probability density function (PDF) of the error. To address this, our study introduces a novel approach, termed the fast minimum error entropy (FMEE) algorithm, aimed at mitigating computational complexity through the utilization of polynomial expansions of the error PDF. Initially, the PDF approximation of a random variable is derived via the Gram–Charlier expansion. Subsequently, we proceed to ascertain and streamline the entropy of the random variable. Following this, the error entropy inherent to the linear regression model is delineated and expressed as a function of the regression coefficient vector. Lastly, leveraging the gradient descent algorithm, we compute the regression coefficient vector corresponding to the minimum error entropy. Theoretical scrutiny reveals that the time complexity of FMEE stands at $O(n)$, in stark contrast to the $O(n^2)$ complexity associated with MEE. Experimentally, our findings underscore the remarkable efficiency gains afforded by FMEE, with time consumption registering less than 1‰ of that observed with MEE. Encouragingly, this efficiency leap is achieved without compromising accuracy, as evidenced by negligible differentials observed between the accuracies of FMEE and MEE. Furthermore, comprehensive regression experiments on real-world electric datasets in northwest China demonstrate that our FMEE outperforms baseline methods by a clear margin.

1. Introduction

Entropy serves as a pivotal metric for quantifying the inherent uncertainty within a system, finding ubiquitous application across a multitude of domains including dimension reduction [1,2], parameter estimation [3,4], identification [5,6], feedback control [7,8], and abnormal detection [9,10,11]. The minimum error entropy (MEE) criterion stands out as a notable framework aimed at minimizing the entropy associated with estimation errors, thereby mitigating uncertainty within the estimation model. Particularly pertinent to linear regression estimation quandaries, MEE diverges from conventional least square methodologies by not only considering the variance of prediction errors, but also incorporating higher-order cumulants, rendering it adept at handling non-Gaussian noise distributions [12,13]. Given the prevalence of non-Gaussian noise in real-world scenarios, the efficacy of MEE has been duly validated across various applications, encompassing adaptive filtering [14,15,16], face recognition [17,18], sparse system identification [19,20,21], stochastic control systems [22,23], and visible light communication [24].

Furthermore, the convergence properties of MEE have been meticulously examined in the prior literature [25,26,27]. In-depth analyses delved into the convergence dynamics of fixed-point MEE methodologies [28], while investigations into the interplay between MEE and Minimum Mean-Squared Error (MMSE) [29], as well as its relationship with maximum correntropy [30], have been extensively explored. Noteworthy extensions to the conventional MEE framework include the proposal of kernel MEE variants [31], elucidations on regularization strategies for MEE implementations [32], and the development of semi-supervised and distributed adaptations [33]. Moreover, within the errors-in-variables (EIV) modeling realm, robustness analyses of MEE methodologies have been expounded upon [34], alongside the proposition of novel methodologies such as the minimum total error entropy approach [35].

The computational demands associated with MEE criterion stem from its time complexity, which scales quadratically as $O(n^2)$. Central to MEE’s computational load is the necessity for a double summation operation during the calculation of the gradient pertaining to the error probability density function (PDF), particularly when employing Parzen windowing [36] and certain types of Kernel functions. Moreover, the selection of an inappropriate Kernel function or its parameters can detrimentally impact the accuracy of the resulting PDF. To solve the problem, efforts have been made to optimize the gradient descent MEE approach, as evidenced by the normalization of the step size in [37], leveraging the power of the input entropy. Additionally, innovative methodologies such as the utilization of a quantization operator, as proposed in [38], have been instrumental in mitigating computational complexity. By mapping the error onto a series of real-valued code words, this approach effectively transforms the double summation operation into a singular form, consequently reducing the algorithm’s computational overhead.

This paper presents a novel approach, the fast minimum error entropy estimation (FMEE) algorithm, tailored specifically for linear regression tasks, leveraging the polynomial expansion technique applied to the error Probability Density Function (PDF). In contrast to the traditional minimum error entropy (MEE) methodology employing Parzen windowing for error PDF derivation, FMEE adopts the Gram–Charlier expansion method to approximate the error PDF, resulting in a notable reduction in time complexity from $O(n^2)$ to $O(n)$. Notably, FMEE obviates the need for Kernel functions or their associated parameters. The proposed algorithm entails several key steps: firstly, the derivation of the PDF approximation for a random variable via the Gram–Charlier expansion, followed by the simplification of the entropy of the random variable facilitated by the orthogonality of the Hermite polynomials. Subsequently, the error of the linear regression model is ascertained, with the error entropy expressed as a function of the regression coefficient vector. Ultimately, the gradient descent technique is employed to deduce the regression coefficient vector corresponding to the minimum error entropy. Experimental validation underscores the efficacy of FMEE, revealing a remarkable reduction in time consumption, with a disparity of less than 1‰ compared to that of the MEE approach for identical problem settings. Importantly, minimal discrepancies are observed between the accuracy levels of FMEE and MEE methodologies.

The rest of the paper is organized as follows: Section 2 introduces the related algorithms, FMEE is proposed in detail in Section 3, Section 4 contains the experiments, and the conclusions are in Section 5.

2. The Related Algorithms

2.1. MEE

MEE minimizes the information contained in the error and maximizes the information captured by the estimated model. Usually, the second-order Renyi entropy [39] is used to measure the entropy contained in the error of the model.

$$H(E)=-logEp(p_E)=-log\int p_E^2(e)de,$$

(1)

where E is the error random variable of MEE, and $H(E)$ is the entropy of E and $p_E(e)$ donates the PDF of E. In this paper, $Ep(·)$ donates the expectation in order to avoid misunderstanding. Then, the PDF of E can be estimated by Parzen windowing:

$${\widehat{p}}_E(e)={\displaystyle \frac{1}{n}}\sum _{i=1}^n{K}_h(e-e_i),$$

(2)

where n is the number of the observed data, K is the kernel function, and h is the bandwidth. Usually, the Gaussian kernel is chosen, and $K_h(z)={\displaystyle \frac{1}{\sqrt{2\pi }h}}exp(-{\displaystyle \frac{z^2}{2h^2}})$. Then, the empirical information of error $\widehat{H}(f)$ can be calculated as:

$$\widehat{H}(f)=-log{\displaystyle \frac{1}{n^2}}\sum _{i=1}^n\sum _{j=1}^n{K}_h(e_i-e_j).$$

(3)

As the logarithmic function is monotone increasing, it can be removed without any influence on the minimizing process. Then, the transformed empirical error information can be obtained as Equation (4):

$$R(e)=-{\displaystyle \frac{1}{n^2}}\sum _{i=1}^n\sum _{j=1}^n{K}_h(e_i-e_j).$$

(4)

For the linear regression, $y={\mathit{w}}^{\mathrm{T}}\mathit{x}+e$, the target of the model is to estimate $\mathit{w}$ from the limited samples. As $e_i=y_i-{\mathit{w}}^{\mathrm{T}}{\mathit{x}}_i$, the corresponding empirical error information is calculated as Equation (5):

$$R\left(\mathit{w}\right)=-{\displaystyle \frac{1}{n^2}}\sum _{i=1}^n\sum _{j=1}^n{K}_h((y_i-{\mathit{w}}^{\mathrm{T}}{\mathit{x}}_i)-(y_j-{\mathit{w}}^{\mathrm{T}}{\mathit{x}}_j)).$$

(5)

Then, the MEE estimator $\widehat{\mathit{w}}$ can be obtained by minimizing $R\left(\mathit{w}\right)$ with gradient descent. The time complexity of MEE is $O\left(n^2\right)$ due to the double summation operation, as shown in Equation (5). As the time consumption of MEE is quadratic to the number of the observed data, the running time of MEE would increase dramatically with the increase in the amount of the observed data.

2.2. The Hermite Polynomials

The Hermite polynomials [40] are a classical orthogonal polynomial sequence. They are defined by the derivatives of the standardized Gaussian PDF $\phi \left(\xi \right)$ as Equation (6):

$${\displaystyle \frac{{\partial }^i\phi \left(\xi \right)}{\partial {\xi }^i}}={(-1)}^i{H}_i\left(\xi \right)\phi \left(\xi \right),$$

(6)

where $H\left(i\right)$ donates the ith-order Hermite polynomial.

The Hermite polynomials forms an orthogonal system, as in Equation (7):

$$\int \phi \left(\xi \right)H_i\left(\xi \right)H_j\left(\xi \right)d\xi =\left\{\begin{array}{cc}i!,\hfill & \mathrm{if}i=j\hfill \\ 0,\hfill & \mathrm{if}i\ne j\hfill \end{array}\right.,$$

(7)

This paper uses the first five Hermite polynomials, shown in Equation (8):

$$\begin{array}{c}{H}_0\left(x\right)=1,\hfill \\ H_1\left(x\right)=x,\hfill \\ H_2\left(x\right)=x^2-1,\hfill \\ H_3\left(x\right)=x^3-3x,\hfill \\ H_4\left(x\right)=x^4-6x^2+3.\hfill \end{array}$$

(8)

2.3. The Gram–Charlier Expansion

The Gram–Charlier expansion obtains the approximation of the PDF of a random variable, and there is no significant loss of accuracy with this expansion, which is demonstrated in [41]. Under the assumption that the random variable is near the normal distribution with the same mean and variance as itself, the Gram–Charlier expansion of the PDF of x can be expressed as:

$$\begin{array}{cc}\hfill p_x\left(\xi \right)\approx & {\widehat{p}}_x\left(\xi \right)={\displaystyle \frac{1}{\sqrt{2\pi }\sigma }}exp\left[-{\displaystyle \frac{{(\xi -\mu )}^2}{2{\sigma }^2}}\right]\ast \hfill \\ & \left[1+{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3\left({\displaystyle \frac{\xi -\mu }{\sigma }}\right)+{\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^4}}{H}_4\left({\displaystyle \frac{\xi -\mu }{\sigma }}\right)\right],\hfill \end{array}$$

(9)

where $\mu$ and $\sigma$ are the mean and standard deviation of x. $\xi$ is the Gaussian random variable whose mean and variable are the same as those of x. ${\kappa }_3\left(x\right)$ and ${\kappa }_4\left(x\right)$ are the third-order and fourth-order cumulant of x, which can also be called skewness and kurtosis. $H_3$ and $H_4$ are the third-order and fourth-order Hermite polynomials.

3. Methodology

This section derives the relation between the differential entropy and the Gram–Charlier expansion of the PDF of the error for linear regression. Moreover, the assumption that the PDF of the regression error is near the Gaussian distribution $\phi \left(\xi \right)$ that has the same mean and variance as the error is implemented.

The differential entropy [42] of a random variable $\mathit{X}$ is shown in Equation (10):

$$H(x)=-\int p(x)logp(x)\mathrm{d}x.$$

(10)

Substitute Equation (9) into Equation (10); the result can be derived in Equation (11):

$$\begin{array}{cc}\hfill H(x)\approx & -\int {\widehat{p}}_x(\xi )log{\widehat{p}}_x(\xi )\mathrm{d}\xi \hfill \\ \hfill =& -\int {\displaystyle \frac{1}{\sqrt{2\pi }\sigma }}exp\left[-{\displaystyle \frac{{(\xi -\mu )}^2}{2{\sigma }^2}}\right]\ast \hfill \\ & \left[1+{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3\left({\displaystyle \frac{\xi -\mu }{\sigma }}\right)+{\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^4}}{H}_4\left({\displaystyle \frac{\xi -\mu }{\sigma }}\right)\right]\ast \hfill \\ \hfill & log[{\displaystyle \frac{1}{\sqrt{2\pi }\sigma }}exp\left[-{\displaystyle \frac{{(\xi -\mu )}^2}{2{\sigma }^2}}\right]\ast \hfill \\ & \left[1+{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3\left({\displaystyle \frac{\xi -\mu }{\sigma }}\right)+{\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^4}}{H}_4\left({\displaystyle \frac{\xi -\mu }{\sigma }}\right)\right]]\mathrm{d}\xi .\hfill \end{array}$$

(11)

This integral is rather difficult to compute. However, notice that if the PDF of x is near the normal density as assumed, it can be inferred that ${\kappa }_3\left(x\right)$ and ${\kappa }_4\left(x\right)$ are very small. Then, the approximation in Equation (12) can be used:

$$log(1+ϵ)\approx ϵ-{\displaystyle \frac{{ϵ}^2}{2}},$$

(12)

where $ϵ$ denotes the infinitesimal.

Then, Equation (11) can be transformed into Equation (13):

$$\begin{array}{cc}\hfill H(x)=& -\int \phi \left(\xi \right)\ast \hfill \\ & \left[1+{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3\left({\displaystyle \frac{\xi -\mu }{\sigma }}\right)+{\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^4}}{H}_4\left({\displaystyle \frac{\xi -\mu }{\sigma }}\right)\right]\ast \hfill \\ & [log\phi \left(\xi \right)+{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3\left({\displaystyle \frac{\xi -\mu }{\sigma }}\right)+\hfill \\ & {\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^4}}{H}_4\left({\displaystyle \frac{\xi -\mu }{\sigma }}\right)-{\displaystyle \frac{1}{2}}({\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3\left({\displaystyle \frac{\xi -\mu }{\sigma }}\right)+\hfill \\ \hfill & {\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^4}}{H}_4\left({\displaystyle \frac{\xi -\mu }{\sigma }}\right){)}^2]\mathrm{d}\xi .\hfill \end{array}$$

(13)

As $H_i$ forms an orthogonal system, Equation (13) can be simplified as Equation (14),

$$H\left(x\right)={\displaystyle \frac{1}{2}}log2\pi \mathrm{e}{\sigma }^2-{\displaystyle \frac{{\kappa }_3^2\left(x\right)}{2\ast 3!{\sigma }^6}}-{\displaystyle \frac{{\kappa }_4^2\left(x\right)}{2\ast 4!{\sigma }^8}}.$$

(14)

The details of the simplification process of Equation (14) are illustrated as follows. This part is to derive Equation (14). First, it is needed to show that

$$\int \phi \left(\xi \right){\left({\displaystyle \frac{\xi -\mu }{\sigma }}\right)}^2{H}_3({\displaystyle \frac{\xi -\mu }{\sigma }})\mathrm{d}\xi =0$$

(15)

and

$$\int \phi \left(\xi \right){({\displaystyle \frac{\xi -\mu }{\sigma }})}^2{H}_4({\displaystyle \frac{\xi -\mu }{\sigma }})\mathrm{d}\xi =0$$

(16)

For Equation (15), suppose that $t=(\xi -\mu )/\sigma$,

$$\begin{array}{cc}\hfill \int & \phi \left(\xi \right){({\displaystyle \frac{\xi -\mu }{\sigma }})}^2{H}_3({\displaystyle \frac{\xi -\mu }{\sigma }})\mathrm{d}\xi \hfill \\ & =\int {\displaystyle \frac{1}{\sqrt{2\pi }\sigma }}exp\left[-{\displaystyle \frac{{(\xi -\mu )}^2}{2{\sigma }^2}}\right]\left[{({\displaystyle \frac{\xi -\mu }{\sigma }})}^3-3({\displaystyle \frac{\xi -\mu }{\sigma }})\right]\mathrm{d}\xi \hfill \\ \hfill & =\int {\displaystyle \frac{1}{\sqrt{2\pi }}}{t}^{2}exp(-{\displaystyle \frac{t^2}{2}})(t^3-3t)\mathrm{d}t\hfill \end{array}$$

(17)

As $exp(-{\displaystyle \frac{t^2}{2}})$ is an even function and $t^5-3t^3$ is an odd function, then

$$\int {\displaystyle \frac{1}{\sqrt{2\pi }}}{t}^{2}exp(-{\displaystyle \frac{t^2}{2}})(t^3-3t)\mathrm{d}t=0$$

(18)

and Equation (15) holds.

For Equation (16), suppose that $t=(\xi -\mu )/\sigma$,

$$\begin{array}{cc}\hfill \int \phi (\xi )& {({\displaystyle \frac{\xi -\mu }{\sigma }})}^2{H}_4({\displaystyle \frac{\xi -\mu }{\sigma }})\mathrm{d}\xi \hfill \\ \hfill =& \int {\displaystyle \frac{1}{\sqrt{2\pi }\sigma }}exp\left[-{\displaystyle \frac{{(\xi -\mu )}^2}{2{\sigma }^2}}\right]\ast \hfill \\ & \left[{({\displaystyle \frac{\xi -\mu }{\sigma }})}^6-6{({\displaystyle \frac{\xi -\mu }{\sigma }})}^4+3{({\displaystyle \frac{\xi -\mu }{\sigma }})}^2\right]\mathrm{d}\xi \hfill \\ \hfill =& \int {\displaystyle \frac{1}{\sqrt{2\pi }}}exp(-{\displaystyle \frac{t^2}{2}})(t^6-6t^4+3t^2)\mathrm{d}t\hfill \\ \hfill =& 3\int {\displaystyle \frac{1}{\sqrt{2\pi }}}exp(-{\displaystyle \frac{t^2}{2}})t^2\mathrm{d}t-6\int {\displaystyle \frac{1}{\sqrt{2\pi }}}exp(-{\displaystyle \frac{t^2}{2}})t^4\mathrm{d}t\hfill \\ & +\int {\displaystyle \frac{1}{\sqrt{2\pi }}}exp(-{\displaystyle \frac{t^2}{2}})t^6\mathrm{d}t\hfill \end{array}$$

(19)

The first term in Equation (19) can be calculated as Equation (20),

$$\begin{array}{cc}\hfill \int {\displaystyle \frac{1}{\sqrt{2\pi }}}& exp(-{\displaystyle \frac{t^2}{2}})t^2\mathrm{d}t\hfill \\ & =\int {\displaystyle \frac{1}{\sqrt{2\pi }}}(-t)\mathrm{d}\left(exp(-{\displaystyle \frac{t^2}{2}})\right)\hfill \\ & ={\displaystyle \frac{1}{\sqrt{2\pi }}}(-t)\left(exp(-{\displaystyle \frac{t^2}{2}})\right){|}_{-\infty }^{+\infty }+\hfill \\ & \int {\displaystyle \frac{1}{\sqrt{2\pi }}}exp(-{\displaystyle \frac{t^2}{2}})\mathrm{d}t\hfill \\ & =1\hfill \end{array}$$

(20)

The second term in Equation (19) can be calculated as Equation (21),

$$\begin{array}{cc}\hfill \int {\displaystyle \frac{1}{\sqrt{2\pi }}}& exp(-{\displaystyle \frac{t^2}{2}})t^4\mathrm{d}t\hfill \\ & =\int {\displaystyle \frac{1}{\sqrt{2\pi }}}(-t^3)\mathrm{d}\left(exp(-{\displaystyle \frac{t^2}{2}})\right)\hfill \\ & ={\displaystyle \frac{1}{\sqrt{2\pi }}}(-t^3)\left(exp(-{\displaystyle \frac{t^2}{2}})\right){|}_{-\infty }^{+\infty }+\hfill \\ & 3\int {\displaystyle \frac{1}{\sqrt{2\pi }}}exp(-{\displaystyle \frac{t^2}{2}})t^2\mathrm{d}t\hfill \\ & =3\hfill \end{array}$$

(21)

The last term in Equation (19) can be calculated as Equation (22),

$$\begin{array}{cc}\hfill \int {\displaystyle \frac{1}{\sqrt{2\pi }}}& exp(-{\displaystyle \frac{t^2}{2}})t^6\mathrm{d}t\hfill \\ & =\int {\displaystyle \frac{1}{\sqrt{2\pi }}}(-t^5)\mathrm{d}\left(exp(-{\displaystyle \frac{t^2}{2}})\right)\hfill \\ & ={\displaystyle \frac{1}{\sqrt{2\pi }}}(-t^5)\left(exp(-{\displaystyle \frac{t^2}{2}})\right){|}_{-\infty }^{+\infty }+\hfill \\ & 5\int {\displaystyle \frac{1}{\sqrt{2\pi }}}exp(-{\displaystyle \frac{t^2}{2}})t^4\mathrm{d}t\hfill \\ & =15\hfill \end{array}$$

(22)

According to Equations (20)–(22),

$$\int \phi \left(\xi \right){({\displaystyle \frac{\xi -\mu }{\sigma }})}^2{H}_4({\displaystyle \frac{\xi -\mu }{\sigma }})\mathrm{d}\xi =15-6\ast 3+3=0$$

(23)

and Equation (16) holds.

Then, for Equation (13),

$$\begin{array}{cc}\hfill H\left(x\right)& =-\int \phi (\xi )log\phi (\xi )\mathrm{d}\xi -\hfill \\ & \int \phi \left(\xi \right)\left[{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3({\displaystyle \frac{\xi -\mu }{\sigma }})+{\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^4}}{H}_4({\displaystyle \frac{\xi -\mu }{\sigma }})\right]\mathrm{d}\xi +\hfill \\ \hfill & \int {\displaystyle \frac{1}{2}}\phi \left(\xi \right){\left[{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3({\displaystyle \frac{\xi -\mu }{\sigma }})+{\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^4}}{H}_4({\displaystyle \frac{\xi -\mu }{\sigma }})\right]}^2\mathrm{d}\xi -\hfill \\ & \int \phi \left(\xi \right)log\phi \left(\xi \right)[{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3({\displaystyle \frac{\xi -\mu }{\sigma }})+\hfill \\ & {\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^4}}{H}_4({\displaystyle \frac{\xi -\mu }{\sigma }})]\mathrm{d}\xi \hfill \\ \hfill & -\int \phi \left(\xi \right)[{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3({\displaystyle \frac{\xi -\mu }{\sigma }})+\hfill \\ & {\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^4}}{H}_4({\displaystyle \frac{\xi -\mu }{\sigma }}){]}^2\mathrm{d}\xi +\hfill \\ & \int {\displaystyle \frac{1}{2}}\phi \left(\xi \right){\left[{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3({\displaystyle \frac{\xi -\mu }{\sigma }})+{\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^4}}{H}_4({\displaystyle \frac{\xi -\mu }{\sigma }})\right]}^3\mathrm{d}\xi \hfill \end{array}$$

(24)

Notice that $H_0=1$, then the second term of Equation (24) is 0. As it is assumed that x is close to a normal random variable, ${\kappa }_3\left(x\right)$ and ${\kappa }_4\left(x\right)$ are very small. Then, the last term of Equation (24) can be omitted as 0, as it involves third-order monomials, which are infinitely smaller than the terms only containing second-order monomials. Then, for Equation (24), take Equations (15) and (16) into consideration, and notice again that $H_0=1$,

$$\begin{array}{cc}\hfill H\left(x\right)& =H\left(\xi \right)-\int {\displaystyle \frac{1}{2}}\phi \left(\xi \right)[{\left[{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3({\displaystyle \frac{\xi -\mu }{\sigma }})\right]}^2+\hfill \\ & {\left[{\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^4}}{H}_4({\displaystyle \frac{\xi -\mu }{\sigma }})\right]}^2+\hfill \\ & 2{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3({\displaystyle \frac{\xi -\mu }{\sigma }}){\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^4}}{H}_4({\displaystyle \frac{\xi -\mu }{\sigma }})]\mathrm{d}\xi \hfill \\ \hfill & -\int \phi \left(\xi \right)\left[log{\displaystyle \frac{1}{\sqrt{2\pi }\sigma }}+logexp(-{\displaystyle \frac{{\xi }^2}{2}})\right]\ast \hfill \\ & \left[{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3({\displaystyle \frac{\xi -\mu }{\sigma }})+{\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^4}}{H}_4({\displaystyle \frac{\xi -\mu }{\sigma }})\right]\mathrm{d}\xi \hfill \\ \hfill =& H\left(\xi \right)-{\displaystyle \frac{{\kappa }_3^2\left(x\right)}{18{\sigma }^6}}-{\displaystyle \frac{{\kappa }_3^2\left(x\right)}{32{\sigma }^8}}-\hfill \\ \hfill & log{\displaystyle \frac{1}{\sqrt{2\pi }\sigma }}\int \phi \left(\xi \right)[{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3({\displaystyle \frac{\xi -\mu }{\sigma }})+\hfill \\ & {\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^3}}{H}_4({\displaystyle \frac{\xi -\mu }{\sigma }})]\mathrm{d}\xi +{\displaystyle \frac{1}{2ln2}}\int \phi \left(\xi \right){\xi }^2\ast \hfill \\ & \left[{\displaystyle \frac{{\kappa }_3\left(x\right)}{3!{\sigma }^3}}{H}_3({\displaystyle \frac{\xi -\mu }{\sigma }})+{\displaystyle \frac{{\kappa }_4\left(x\right)}{4!{\sigma }^3}}{H}_4({\displaystyle \frac{\xi -\mu }{\sigma }})\right]\mathrm{d}\xi \hfill \\ \hfill =& {\displaystyle \frac{1}{2}}log2\pi \mathrm{e}{\sigma }^2-{\displaystyle \frac{{\kappa }_3^2\left(x\right)}{18{\sigma }^6}}-{\displaystyle \frac{{\kappa }_4^2\left(x\right)}{32{\sigma }^8}}\hfill \end{array}$$

(25)

So, Equation (14) holds.

For the linear regression, $e_i=y_i-w^{\mathrm{T}}{\mathit{x}}_i$, the standard deviation of the error $e_w$ is in Equation (26):

$${\sigma }^2\left(e_w\right)=Ep\left(E_w^2\right)-{\left[Ep\left(E_w\right)\right]}^2=Ep\left(E_w^2\right)-{(\overline{y}-{\mathit{w}}^{\mathrm{T}}\overline{\mathit{x}})}^2,$$

(26)

where $\overline{y}$ is the mean of y and $\overline{\mathit{x}}$ donates the mean of $\mathit{x}$.

To simplify the calculation of ${\kappa }_4\left(e_w\right)$, let $e_w$ become a random variable with 0 mean:

$$e_{w_i}=y_i-{\mathit{w}}^{\mathrm{T}}{\mathit{x}}_i-\overline{y}+{\mathit{w}}^{\mathrm{T}}\overline{\mathit{x}}.$$

(27)

Notice that the transformation in Equation (27) does not change the standard deviation and the entropy of $e_w$ as ${\sigma }^2(e+c)={\sigma }^2\left(e\right)$ and $H(e+c)=H\left(e\right)$ when c is a constant number. In the rest of the paper, to keep clarity, ${\sigma }^2(·)$ is replaced with $var(·)$.

Then, for linear regression, its entropy of the error can be expressed as Equation (28):

$$\begin{array}{cc}\hfill H\left(e_w\right)=& {\displaystyle \frac{1}{2}}log2\pi \mathrm{e}+{\displaystyle \frac{1}{2}}logvar\left(e_w\right)\hfill \\ & -{\displaystyle \frac{{\kappa }_3^2\left(e_w\right)}{2\ast 3!va{r}^3\left(e_w\right)}}-{\displaystyle \frac{{\kappa }_4^2\left(e_w\right)}{2\ast 4!va{r}^4\left(e_w\right)}},\hfill \end{array}$$

(28)

where

$${\kappa }_3\left(e_w\right)=Ep\left[{(y-{\mathit{w}}^{\mathrm{T}}\mathit{x}-\overline{y}+{\mathit{w}}^{\mathrm{T}}\overline{\mathit{x}})}^3\right],$$

(29)

$${\kappa }_4\left(e_w\right)=Ep\left[{(y-{\mathit{w}}^{\mathrm{T}}\mathit{x}-\overline{y}+{\mathit{w}}^{\mathrm{T}}\overline{\mathit{x}})}^4\right]-3.$$

(30)

To minimize $H\left(e_w\right)$, the derivative of $H\left(e_w\right)$ with respect to w is calculated in Equation (31):

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial H\left(e_w\right)}{\partial \mathit{w}}}=& {\displaystyle \frac{1}{2var\left(e_w\right)ln2}}{\displaystyle \frac{\partial var\left(e_w\right)}{\partial \mathit{w}}}-{\displaystyle \frac{1}{2\ast 3!va{r}^6\left(e_w\right)}}\ast \hfill \\ & [2va{r}^3\left(e_w\right){\kappa }_3\left(e_w\right){\displaystyle \frac{\partial {\kappa }_3\left(e_w\right)}{\partial \mathit{w}}}-\hfill \\ & 3{\kappa }_3^2\left(e_w\right)va{r}^2\left(e_w\right){\displaystyle \frac{\partial var\left(e_w\right)}{\partial \mathit{w}}}]\hfill \\ \hfill -& {\displaystyle \frac{1}{2\ast 4!va{r}^8\left(e_w\right)}}[2va{r}^4\left(e_w\right){\kappa }_4\left(e_w\right){\displaystyle \frac{\partial {\kappa }_4\left(e_w\right)}{\partial \mathit{w}}}-\hfill \\ & 4{\kappa }_4^2\left(e_w\right)va{r}^3\left(e_w\right){\displaystyle \frac{\partial var\left(e_w\right)}{\partial \mathit{w}}}],\hfill \end{array}$$

(31)

where

$${\displaystyle \frac{\partial var\left(e_w\right)}{\partial \mathit{w}}}=-2Ep\left[(y-{\mathit{w}}^{\mathrm{T}}\mathit{x})\mathit{x}\right]+2(\overline{y}-{\mathit{w}}^{\mathrm{T}}\overline{\mathit{x}})\overline{\mathit{x}},$$

(32)

$${\displaystyle \frac{\partial {\kappa }_3\left(e_w\right)}{\partial \mathit{w}}}=3Ep\left[{(y-{\mathit{w}}^{\mathrm{T}}\mathit{x}-\overline{y}+{\mathit{w}}^{\mathrm{T}}\overline{\mathit{x}})}^2(-\mathit{x}+\overline{\mathit{x}})\right],$$

(33)

$${\displaystyle \frac{\partial {\kappa }_4\left(e_w\right)}{\partial \mathit{w}}}=4Ep\left[{(y-{\mathit{w}}^{\mathrm{T}}\mathit{x}-\overline{y}+{\mathit{w}}^{\mathrm{T}}\overline{\mathit{x}})}^3(-\mathit{x}+\overline{\mathit{x}})\right].$$

(34)

Then, the optimal $\mathit{w}$ for the minimum of $H\left(e_w\right)$ can be obtained by the iteration scheme in Equation (35) with gradient descent:

$${\widehat{\mathit{w}}}_{k+1}={\widehat{\mathit{w}}}_k-\alpha {\displaystyle \frac{\partial H\left(e_{w_k}\right)}{\partial {\widehat{\mathit{w}}}_k}},$$

(35)

where $\alpha$ is the step size determined by Armijo condition [43].

To compute ${\displaystyle \frac{\partial H\left(e_{w_k}\right)}{\partial {\mathit{w}}_k}}$ for FMEE in every iteration, $var\left(e_{w_k}\right)$, ${\kappa }_3\left(e_{w_k}\right)$, ${\kappa }_4\left(e_{w_k}\right)$, ${\displaystyle \frac{\partial var\left(e_{w_k}\right)}{\partial {\mathit{w}}_k}}$, ${\displaystyle \frac{\partial {\kappa }_3\left(e_{w_k}\right)}{\partial {\mathit{w}}_k}}$, and ${\displaystyle \frac{\partial {\kappa }_4\left(e_{w_k}\right)}{\partial {\mathit{w}}_k}}$ are needed. From Equations (26) and (29)–(34), the time complexity of $var\left(e_{w_k}\right)$, ${\kappa }_3\left(e_{w_k}\right)$, ${\kappa }_4\left(e_{w_k}\right)$, ${\displaystyle \frac{\partial var\left(e_{w_k}\right)}{\partial {\mathit{w}}_k}}$, ${\displaystyle \frac{\partial {\kappa }_3\left(e_{w_k}\right)}{\partial {\mathit{w}}_k}}$, and ${\displaystyle \frac{\partial {\kappa }_4\left(e_{w_k}\right)}{\partial {\mathit{w}}_k}}$ is $O\left(n\right)$, respectively. Then, the computational complexity of FMEE is $O\left(n\right)$. As the time complexity of MEE is $O\left(n^2\right)$ due to the double summation operation, FMEE runs faster than MEE does.

4. Experiments

In this section, a comprehensive array of experiments is undertaken, encompassing both numerical simulations and real-world scenarios. The numerical simulations serve to validate the efficacy of the FMEE approach and to scrutinize the time efficiency of both FMEE and MEE methodologies. Concurrently, practical experiments are conducted aimed at forecasting power outages within a city located in northwest China.

Regarding the numerical simulations, we consider a scenario where $\mathit{x}\in {\mathbb{R}}^{10}$, with the model defined as $y=\mathit{w}{\ast }^{\mathrm{T}}\mathit{x}+e$, where $\mathit{w}\ast ={[1,-1,1,-1,1,-1,1,-1,1,-1]}^{\mathrm{T}}$ and $\mathit{x}$ follows a normal distribution $\mathcal{N}(0,I_{10})$. Two distinct types of noise are examined in our experiments. Firstly, Gaussian noise $e\sim \mathcal{N}(0,1)$ is employed, while the second type comprises generalized Gaussian noise characterized by a probability density function $f\left(e\right)\propto exp{(-\alpha |e\left|\right)}^{0.3}$. This heavy-tailed distribution, utilized herein, aligns with previous works, such as references [27,35]. Although FMEE is derived under the assumption of near-Gaussian noise, it is anticipated that FMEE remains effective in the presence of non-Gaussian, heavy-tailed noises. Here, $\alpha$ denotes a constant adjusted to maintain a variance of 1 for e. For MEE, the step size is set to $\sqrt{0.005\pi }$, and the scale parameter for the Gaussian kernel is 10, mirroring the settings adopted in Reference [27]. Across the experiments, sample sizes range from 100 to 500, with each combination of sample size and noise type repeated 100 times. The ensuing experimental results are detailed below.

Table 1. The running time of MEE and FMEE for Gaussian noise.

Sample Size	Average Running Time		Optimal Running Time
	MEE	FMEE	MEE	FMEE
100	103.89	0.0401	77.77	0.0156
200	384.31	0.0580	310.05	0.0312
300	801.52	0.0769	654.21	0.0468
400	1430.61	0.1000	1254.12	0.0468
500	2306.24	0.1104	2072.50	0.0624

provides a comparative analysis of the time consumption between MEE and FMEE in the presence of Gaussian noise. FMEE demonstrates a notable acceleration in computational efficiency compared to MEE. Specifically, while the computational time of MEE exhibits quadratic growth relative to the sample size, FMEE’s computational overhead scales linearly with the number of samples. Notably, FMEE achieves remarkable speed, processing 500 objects in approximately 0.1 s, thereby positioning it as a highly promising candidate for time-sensitive applications.

Table 2. The mean squared error of MEE and FMEE for Gaussian noise.

Sample Size	Average MSE		Optimal MSE
	MEE	FMEE	MEE	FMEE
100	0.9108	1.4027	0.5697	0.6542
200	0.9311	1.0528	0.6295	0.7258
300	0.9573	1.0190	0.7600	0.7462
400	0.9696	0.9769	0.8024	0.7588
500	0.9604	0.9938	0.8024	0.8302

compares the mean squared error between MEE and FMEE in the context of Gaussian noise. Overall, MEE exhibits superior performance compared to FMEE in terms of MSE. Particularly noteworthy is MEE’s substantial advantage over FMEE, especially evident when the sample size is 100. However, as the sample size increases, the discrepancy in MSE between the two algorithms diminishes. By the time the sample size reaches 500, the difference dwindles to a mere $2\%$. This observation suggests that while FMEE may lag slightly behind MEE, particularly in larger-scale datasets, such disparity remains acceptable in practical applications, given FMEE’s significantly reduced computational time compared to MEE.

Table 3. The number of iterations of MEE and FMEE for Gaussian noise.

Sample Size	Average Iteration Number		Optimal Iteration Number
	MEE	FMEE	MEE	FMEE
100	797.01	38.21	592	6
200	697.09	38.29	592	7
300	656.61	36.88	542	7
400	650.03	40.67	571	9
500	637.03	36.17	586	7

analyses the iteration counts required for convergence between MEE and FMEE in the context of Gaussian noise. Notably, FMEE achieves convergence with significantly fewer iterations compared to MEE. Furthermore, it is observed that the iteration count of MEE gradually decreases as the sample size increases. Conversely, the iteration count of FMEE remains relatively constant, irrespective of variations in sample size.

Table 4. The running time of MEE and FMEE for non-Gaussian noise.

Sample Size	Average Running Time		Optimal Running Time
	MEE	FMEE	MEE	FMEE
100	107.96	0.0413	85.24	0.0156
200	348.66	0.0635	270.26	0.0312
300	800.50	0.0819	727.45	0.0468
400	1431.34	0.1027	1274.87	0.0624
500	2179.33	0.1178	1952.27	0.0624

provides a comparative assessment of the time consumption between MEE and FMEE in the context of non-Gaussian noise. Across all five experimental groups, FMEE consistently demonstrates a remarkable reduction in time consumption, consistently amounting to less than 1‰ of MEE’s time expenditure. Moreover, in accordance with theoretical predictions, the computational cost of MEE exhibits quadratic growth relative to the sample size, whereas FMEE’s time overhead scales linearly with the sample size.

Table 5. The mean squared error of MEE and FMEE for non-Gaussian noise.

Sample Size	Average MSE		Optimal MSE
	MEE	FMEE	MEE	FMEE
100	1.0844	0.9604	0.1584	0.1630
200	0.8485	1.1554	0.2654	0.2607
300	0.9110	1.2314	0.3140	0.2659
400	1.1037	1.0550	0.3672	0.2891
500	0.9536	1.0491	0.4995	0.4668

conducts a comparative evaluation of the mean squared error (MSE) between MEE and FMEE in the context of non-Gaussian noise. Generally, the disparity between the MSE values of FMEE and MEE is relatively minor. Notably, FMEE outperforms MEE when the sample size is 100 or 400, while MEE exhibits superior performance for sample sizes of 200, 300, and 500. Additionally, FMEE attains the optimal MSE across all sample sizes, except for $n=100$. Furthermore, it is observed that the MSE values for non-Gaussian noise, as depicted in Table 5, span a broader range compared to those for Gaussian noise, indicating greater variability in results for both MEE and FMEE.

Table 6. The number of iterations of MEE and FMEE for non-Gaussian noise.

Sample Size	Average Iteration Number		Optimal Iteration Number
	MEE	FMEE	MEE	FMEE
100	782.88	34.33	618	8
200	689.67	35.72	563	9
300	644.95	36.72	584	10
400	648.70	37.55	580	12
500	633.25	35.35	571	12

compares the number of iterations required for convergence between MEE and FMEE in the context of non-Gaussian noise. FMEE consistently demonstrates faster convergence compared to MEE. Furthermore, it is observed that the iteration count of MEE decreases as the sample size increases. In contrast, the iteration count of FMEE remains relatively stable despite variations in sample size.

Based on the findings presented in Table 1, Table 2, Table 3, Table 4, Table 5 and Table 6, it is evident that FMEE consistently yields comparable outcomes to MEE, albeit with significantly reduced time consumption and iteration counts. Notably, FMEE stands out for its remarkable efficiency, being approximately 1000 times faster than MEE and achieving convergence within approximately 0.1 s for 500 instances. Given its linear time complexity of $O\left(n\right)$, FMEE holds considerable promise for deployment in real-time scenarios characterized by non-Gaussian noise.

Subsequently, we embark on practical experimentation utilizing transformer characteristic data to forecast distribution network failures. The transformers are categorized into two groups based on their geographical location within distinct levels of urban activity within a specific city in northwest China. Initially, our attention is directed towards a subset of 1265 transformers situated within a smaller, less densely populated area of the aforementioned city. We undertake a random partitioning of this dataset into two segments: a training set comprising $80\%$ of the data, totaling 1012 instances, and a verification set encompassing the remaining $20\%$, comprising 253 instances. To be more specific, the transformer dataset comprises various characteristic variables, involving the standardized heavy overload duration, maximum active load ratio, average active load ratio, mean three-phase unbalance, standardized heavy three-phase unbalance duration and so forth.

We conduct a comparative analysis, pitting our proposed FMEE against three established baseline methodologies: logistic regression, neural network, and support vector machine. The dataset comprises 1265 transformer characteristic variables associated with substantial load levels, undergoing 30 rounds of random partitioning. Subsequently, each randomly divided dataset is subjected to four distinct algorithms. The predictive efficacy of these algorithms on the verification set is assessed based on F-measure and error rate metrics.

Figure 1 depicts the F-measure evaluations stemming from 30 prediction instances conducted by four distinct algorithms. Notably, each evaluation involves consistent partitioning of the dataset across the four algorithms. A discernible trend emerges wherein all algorithms yield F-measure values surpassing 0.8. Notably, our proposed FMEE exhibits the most promising performance, boasting an average F-measure of 0.904, thereby outshining the other three baseline methodologies by a significant margin.

To facilitate a comprehensive comparison among the four algorithms, Figure 2 depicts the error rate and F-measure of fault outage prediction results in the form of line and box charts. Notably, FMEE emerges as a frontrunner, showcasing a substantial advantage over the sophisticated support vector machine in terms of error rate. Specifically, our FMEE achieves an average error rate of $9.1\%$, surpassing the neural network, logistic regression, and support vector machine by margins of $0.3\%$, $4.2\%$, and $4.3\%$, respectively. These findings underscore the clear superiority of our proposed FMEE methodology.

Furthermore, we extend our experimentation to a more densely populated urban region within the northwest Chinese city. Figure 3 illustrates the F-measure evaluations derived from 30 independent tests conducted across the four comparative algorithms. On a comprehensive scale, our proposed FMEE attains an F-measure of 0.891, surpassing baseline methodologies such as neural networks, logistic regression, and support vector machines by margins of 0.013, 0.023, and 0.028, respectively.

Finally, we report the error rate of F-measure concerning fault outage prediction results derived from four comparative algorithms, represented through line and box charts. As shown in Figure 4, our proposed FMEE achieves an average error rate of $11.2\%$, showcasing a notable superiority over three baseline methodologies. These findings robustly underscore the efficacy of our algorithm in addressing regression problems.

5. Conclusions

This study introduces a rapid minimum error entropy algorithm tailored for linear regression. Several notable conclusions emerge from our investigation. Firstly, the proposed FMEE exhibits significantly faster convergence, requiring fewer iterations compared to MEE. Secondly, while FMEE does not surpass traditional MEE in performance for Gaussian noise and yields comparable results for non-Gaussian noise as measured by mean squared error, its computational efficiency is strikingly superior, consuming less than 1‰ of the time required by MEE. Thirdly, despite being derived under the assumption of noise proximity to Gaussian distribution, FMEE demonstrates efficacy in handling non-Gaussian, heavy-tailed noise distributions. Fourthly, owing to its exceptional time efficiency, FMEE holds promise for real-time applications in scenarios featuring non-Gaussian noise. Moving forward, further analysis of FMEE’s convergence conditions and exploration of noise approximation methods for arbitrary distributions represent avenues for future research.