Statistical Modeling of Implicit Functional Relations

Abstract

This study considers the statistical estimation of relations presented by implicit functions. Such structures define mutual interconnections of variables rather than outcome variable dependence by predictor variables considered in regular regression analysis. For a simple case of two variables, pairwise regression modeling produces two different lines of each variable dependence using another variable, but building an implicit relation yields one invertible model composed of two simple regressions. Modeling an implicit linear relation for multiple variables can be expressed as a generalized eigenproblem of the covariance matrix of the variables in the metric of the covariance matrix of their errors. For unknown errors, this work describes their estimation by the residual errors of each variable in its regression by the other predictors. Then, the generalized eigenproblem can be reduced to the diagonalization of a special matrix built from the variables’ covariance matrix and its inversion. Numerical examples demonstrate the eigenvector solution’s good properties for building a unique equation of the relations between all variables. The proposed approach can be useful in practical regression modeling with all variables containing unobserved errors, which is a common situation for the applied problems.

1. Introduction

This paper describes several approaches to the statistical estimation of relations expressed via implicit functions. In regular regression analysis, an outcome variable is observed or measured with errors, and it is presented as an explicit function of the other variables, which are free from errors. The minimization of squared errors by the outcome variable produces an estimation of the parameters in ordinary least-squares (OLS) regression [1]. Depending on which variable is taken as the outcome variable, it is possible to build its own model based on the other variables. These models are different and cannot be reduced from one to another. Even in the simple case of two variables, y and x, in their linear dependence, there are two solutions corresponding to minimizations of deviations in the y-axis or x-axis directions, respectively. These models can be presented graphically as two different lines of each variable dependence by another variable, intersecting at the point of the variables’ mean values. These two relations are not inversible, and only in the case of correlation |r| = 1 do the models coincide. The same two OLS solutions correspond to another scheme of statistical modeling when two random variables satisfy the binormal distribution, and each regression can be obtained as the conditional expectation of one variable by the fixed values of the other variable.

In a general case of n variables, there are n different linear regressions of each variable formed by all the others. These models are not inversible, and thus cannot be reduced from one to another. Such a situation raises a question—is it possible to build a unique model of all the variables connected in one relationship and to make it inversible to express any needed variable as a function of all other variables? A reasonable approach for simultaneous fitting by all directions consists of the minimization of the shortest distances or the length of perpendiculars from the observations to the line, plane, or surface of the variables’ connection in their theoretical model. The first results on such modeling were probably obtained by the American statistician Robert Adcock [2,3,4], and their development led to orthogonal regression by any number of variables, proposed by Karl Pearson [5] as the eigenvector with the minimum eigenvalue of the correlation matrix. In contrast to OLS regression, orthogonal regression is not invariant to the scaling transformation; therefore, it can be applied only to the variables of the same units or to the standardized variables, and its properties have been studied further in many works [6,7,8].

If a functional relation, for example, a physics law or an economics supply–demand link, or any other kind of mutual interconnection, can be assumed for variables measured with errors, then a more general approach of the maximum likelihood objective can be applied to building the models. Such models are called deterministic functional relations (notably, this does not concern functional data analyses, which deal with curves, surfaces, or other structures varying over time). For linear or linearized functional relations, such an approach produces a generalized eigenproblem with two covariance matrices—one of the observed variables and one of the errors of observations by the variables [9]. For uncorrelated errors with equal variances, the generalized eigenproblem reduces to the regular eigenproblem for orthogonal regression.

In the simple case of two variables, if the errors’ variances are proportional to the variances of the variables themselves, the result corresponds to the so-called diagonal regression, which can be presented graphically as a diagonal line of the first and third, or otherwise the second and fourth, quadrants in the plane of the standardized variables [10]. Diagonal regression was proposed by Ragnar Frisch, a Nobel laureate in economics, in his work on confluence analysis, in which functional structure relations were built for random or non-random variables observed with additional random noise [11]. In contrast to OLS regression, for which the slope depends on the pair correlation r, the diagonal regression of the standardized variables depends only on the sign of r, and similar constructs have been described for models using two predictors [12,13].

Diagonal regression is also known as geometric mean regression, standard (reduced) major axis regression, and other names, as described in an extensive review [14]. More references on orthogonal and diagonal regressions and related problems can be found in [15,16,17,18,19,20,21,22,23,24]. The so-called total least squares of the errors-in-variables regression and various other techniques have been developed as well [25,26,27,28,29,30,31].

The current paper considers regression modeling for the functional relations defined in an implicit form. For two variables, the explicit dependence of one variable on another can be written as y = f(x) or x = g(y), while the implicit function can be presented as F(x, y) = 0. In many cases, it is more convenient to use the implicit form in statistical modeling. For example, if the theoretical relation is assumed to be a circle x² + y² = ρ² in the center of coordinates with the radius ρ, it is easier to use two new variables, u₁ = x² and u₂ = y², and to build one simple linear model, u₁ + u₂ =k, where the estimated constant will then be used to find the radius as ρ = k^0.5. Otherwise, initially transforming the model to an explicit form, we would have a more complicated problem of splitting the data to fit each portion to one of the two branches of the functions y₁ = +(ρ² − x²)^0.5 and y₂ = −(ρ² − x²)^0.5 and then to combine the estimations into one value for the radius. A similar problem of restoring a Pythagorean linear relation between the squares of three variables can be considered by the measurement of three sides of right triangles. The well-known ideal gas law PV/T = constant can be restored from data on the pressure, volume, and temperature, which is modeled in the general linear equation after the logarithmic transformation of this relation.

The implicit function helps to evaluate data corresponding to non-unique but possible multi-value dependencies. The implicit function of the second-order curve of conical sections given by two variables in the linear, quadratic, and mixed items was applied for statistical modeling in [32]. It helps to build a model of two possible values of y at one given value of x, as observed, for instance, in a parabola function extending at some angle to the x-axis. An implicit relations approach was applied for modeling the unitary response data useful for the estimation of shares in the total, and for Thurstone scaling data [33]. Building an implicit logistic regression for description of the so-called supercritical pitchfork bifurcation known in chaotic systems was used in [34]. Bifurcation is a function behavior where a number of solutions and their structure can change abruptly, and such modeling can be helpful in dealing with messy data characterized by a wide range of response variables at each point of the predictors’ values, for instance, those known in advertising and marketing research. Various other applications of the implicit multi-value functions are also known [35,36,37,38,39]. Data fitting for implicit functions can be performed by various techniques of nonlinear statistical estimation available in modern software packages. It is also often possible to use new variables in which the original implicit function can be transformed to a linear function, similar to reducing a circle to the line in the coordinates squared from the original variables.

The regression model of an implicit linear relation for multiple variables can be expressed as the generalized eigenproblem containing two covariance matrices—of the variables and of the errors of their observations. As shown in this study, in the common case of unknown errors, these can be estimated by the residual errors in each variable linear regression by the other predictors. Then, the generalized eigenproblem is reduced to diagonalization of the special matrix constructed from the covariance matrix of the variables and its inversion. The proposed approach enables building a unique equation of relations between all variables when they contain unobserved errors.

The paper is structured as follows: Section 2 describes a case of two variables in different modeling approaches; Section 3 extends these techniques to multiple variables; Section 4 discusses numerical examples; and Section 5 summarizes.

2. Modeling for Two Variables

Let us briefly consider OLS regressions. As it is assumed in applied regression modeling, the outcome variable is observed or measured with errors and depends on predictors which are free from errors. For a simple case of two variables, y and x, when a theoretical unobserved value of the dependent variable z₂ is linearly connected to x, the model can be written as

$$y_i=z_{2i}+{\epsilon }_i=a{x}_i+b+{\epsilon }_i$$

(1)

where i denotes observations (i = 1, 2, …, N, where N—their total number), a and b are unknown parameters, and $\epsilon$ is the error by y. The OLS minimizing the sum of squared deviations in Equation (1) yields the solution for the regression of y by x:

$$y=m_y+r\frac{{\sigma }_y}{{\sigma }_x}(x-m_x)$$

(2)

where the sample mean values are denoted as m_x and m_y, the standard deviations are ${\sigma }_x$ and ${\sigma }_y$, and r is the correlation between the variables.

Similarly, assuming errors in the x variable, when a theoretical unobserved dependent variable z₁ is linearly related to y, we obtain another model:

$$x_i=z_{1i}+{\delta }_i=c{y}_i+d+{\delta }_i$$

(3)

where c and d are unknown parameters, and $\delta$ is the error by x. The OLS criterion for the total of squared deviations in (3) leads to the regression of x by y:

$$x=m_x+r\frac{{\sigma }_x}{{\sigma }_y}(y-m_y)$$

(4)

The two solutions, (2) and (4), correspond to minimizations of deviations in the y-axis or x-axis directions, respectively, and they can be presented graphically as two different lines of each variable dependence by another one. The lines intersect at the point (m_x, m_y) of mean values, and (4) inclines more steeply than (2) to the x-axis, because the slope of the line (2) is proportional to the correlation |r| < 1, and the slope of the line (4) is proportional to the inverse value |1/r| > 1. Only in the case of correlation reaching one by the absolute value, |r| = 1, do both these equations coincide. If y-m_y is expressed by x-m_x from (4), it would not coincide with (2) because there will be the term 1/r instead of r, as in (2), but taking the geometric mean of the slopes of these equations yields the expression

$$y=m_y+sgn\left(r\right)\frac{{\sigma }_y}{{\sigma }_x}(x-m_x)$$

(5)

which does not depend on the value of the correlation, but only on its sign sgn(r). Building this model explains its name of “geometric mean regression”, and it presents a compromise between two OLS solutions, is symmetric by both variables, and can be inverted. The coordinates of the standardized variables (centered and normalized by the standard deviations) graphically present a diagonal line with the slope sgn(r), which explains the name of “diagonal regression”.

In addition to classical regression modeling, the relationship models between both variables observed with errors have been considered, with their theoretical values z₁ and z₂ satisfying the general linear equation of their connection:

$${\alpha }_0+z_{1i}{\alpha }_1+z_{2i}{\alpha }_2=0$$

(6)

where the alphas are unknown parameters. Using this general equation for both variables observed with errors (1) and (3), and taking into account the identical relation (6), yields the relation:

$${\alpha }_0+x_i{\alpha }_1+y_i{\alpha }_2={\alpha }_0+\left(z_{1i}+{\delta }_i\right){\alpha }_1+\left(z_{2i}+{\epsilon }_i\right){\alpha }_2={\delta }_i{\alpha }_1+{\epsilon }_i{\alpha }_1,$$

(7)

and the errors by each variable are assumed to be of zero mean value. To simplify the derivation, let the variables be centered, then the intercept ${\alpha }_0$ on the left-hand side of (7) can be omitted, and this side in matrix form can be written as $\overrightarrow{x}{\alpha }_1+\overrightarrow{y}{\alpha }_2$ where $\overrightarrow{x}$ and $\overrightarrow{y}$ are the column-vectors of N observations by the variables. Then, the scalar product of this total vector by itself yields the quadratic form:

$${\left(\overrightarrow{x}{\alpha }_1+\overrightarrow{y}{\alpha }_2\right)}^{\prime }\left(\overrightarrow{x}{\alpha }_1+\overrightarrow{y}{\alpha }_2\right)={\left(\begin{array}{c}{\alpha }_1\\ {\alpha }_2\end{array}\right)}^{\prime }\left(\begin{array}{cc}{c}_{xx}& c_{xy}\\ c_{yx}& c_{yy}\end{array}\right)\left(\begin{array}{c}{\alpha }_1\\ {\alpha }_2\end{array}\right)=\alpha \prime C\alpha$$

(8)

where prime denotes transposition, c_xx and c_yy are the second moments, or sample variances for the variables x and y, respectively, and c_xy (equal c_yx) is their sample covariance. The second-order matrix C on the right-hand side of (8) is the variance–covariance matrix of the variables, and the vector $\alpha$ contains both alpha parameters of the model. Similarly, the right-hand side of (7) in the matrix form can be written as $\overrightarrow{\delta }{\alpha }_1+\overrightarrow{\epsilon }{\alpha }_2$, where the column-vectors contain N errors by the observed variables. The scalar product of this total vector of errors by itself yields another quadratic form:

$${\left(\overrightarrow{\delta }{\alpha }_1+\overrightarrow{\epsilon }{\alpha }_2\right)}^{\prime }\left(\overrightarrow{\delta }{\alpha }_1+\overrightarrow{\epsilon }{\alpha }_2\right)={\left(\begin{array}{c}{\alpha }_1\\ {\alpha }_2\end{array}\right)}^{\prime }\left(\begin{array}{cc}{s}_{xx}& s_{xy}\\ s_{yx}& s_{yy}\end{array}\right)\left(\begin{array}{c}{\alpha }_1\\ {\alpha }_2\end{array}\right)=\alpha \prime S\alpha$$

(9)

where s_xx and s_yy are the second moments, or sample variances for the errors by the variables x and y, respectively, and s_xy (or s_yx) is the sample covariance of the errors. Thus, the second-order matrix S on the right-hand side of (9) is the variance–covariance matrix of the variables’ errors.

Finding the minimum sum of squares of the observations’ deviations from the model on the left-hand side of (7) subject to the fixed value at its right-hand side (needed for the parameters identifiability) can be presented as the Rayleigh quotient of two quadratic forms ${\alpha }^{\prime }C\alpha /\alpha \prime S\alpha$. Minimization of this ratio can be reduced to the conditional least squares problem:

$$L={\alpha }^{\prime }C\alpha -\lambda ({\alpha }^{\prime }S\alpha -1)$$

(10)

where $\lambda$ is the Lagrange term. The derivative of (10) with vector $\alpha$ set to zero for finding the extrema yields the relation:

$$C\alpha =\lambda S\alpha$$

(11)

which is the generalized eigenproblem.

In practical modeling, it is commonly assumed that the errors by different variables are independent, so their covariance equals zero, s_xy = 0, and S is the diagonal matrix. Then, problem (11) reduces to the regular eigenproblem:

$$\left(\begin{array}{cc}{c}_{xx}/s_{xx}& c_{xy}/s_{xx}\\ c_{yx}/s_{yy}& c_{yy}/s_{yy}\end{array}\right)\left(\begin{array}{c}{\alpha }_1\\ {\alpha }_2\end{array}\right)=\lambda \left(\begin{array}{c}{\alpha }_1\\ {\alpha }_2\end{array}\right)$$

(12)

The characteristic equation for problem (12) is defined by the zero value of the determinant:

$$\left|\begin{array}{cc}\frac{c_{xx}}{s_{xx}}-\lambda & \frac{c_{xy}}{s_{xx}}\\ \frac{c_{yx}}{s_{yy}}& \frac{c_{yy}}{s_{yy}}-\lambda \end{array}\right|={\lambda }^2-\left(\frac{c_{xx}}{s_{xx}}+\frac{c_{yy}}{s_{yy}}\right)\lambda +\left(\frac{c_{xx}}{s_{xx}}\cdot \frac{c_{yy}}{s_{yy}}-\frac{c_{xy}}{s_{xx}}\cdot \frac{c_{yx}}{s_{yy}}\right)=0$$

(13)

Solving this quadratic equation and taking the minimum root corresponding to the minimization of the objective (10) produces the expression:

$${\lambda }_{min}=\frac{1}{2}\left(\frac{c_{xx}}{s_{xx}}+\frac{c_{yy}}{s_{yy}}\right)-\sqrt{\frac{1}{4}{\left(\frac{c_{xx}}{s_{xx}}+\frac{c_{yy}}{s_{yy}}\right)}^2-\frac{c_{xx}{c}_{yy}-c_{xy}^2}{s_{xx}{s}_{yy}}}$$

(14)

With the minimum eigenvalue (14), the eigenvector can be found. Substituting (14) into one of the equations of the homogeneous system (12), for example, the first one, yields:

$$\left(\frac{c_{xx}}{s_{xx}}-{\lambda }_{min}\right){\alpha }_1+\frac{c_{xy}}{s_{xx}}{\alpha }_2=0$$

(15)

The elements ${\alpha }_1$ and ${\alpha }_2$ of the eigenvector $\alpha$ (11) are defined up to a constant; thus, for their identifiability, we can fix one of them, for instance, ${\alpha }_2=1$. Then, the expression (15) reduces to

$$\left(\frac{1}{2}\left(\frac{c_{xx}}{s_{xx}}-\frac{c_{yy}}{s_{yy}}\right)+\sqrt{\frac{1}{4}{\left(\frac{c_{xx}}{s_{xx}}-\frac{c_{yy}}{s_{yy}}\right)}^2+\frac{c_{xy}^2}{s_{xx}{s}_{yy}}}\right){\alpha }_1+\frac{c_{xy}}{s_{xx}}=0$$

(16)

with the solution for the unknown element ${\alpha }_1$ of the eigenvector as follows:

$${\alpha }_1=\left(\frac{1}{2}\left(\frac{c_{xx}}{s_{xx}}-\frac{c_{yy}}{s_{yy}}\right)-\sqrt{\frac{1}{4}{\left(\frac{c_{xx}}{s_{xx}}-\frac{c_{yy}}{s_{yy}}\right)}^2+\frac{c_{xy}^2}{s_{xx}{s}_{yy}}}\right)/\left(\frac{c_{xy}}{s_{yy}}\right)$$

(17)

Then, the general linear relation (6) for the centered variables, expressed as the dependence of y by x (for the sake of comparison with the models (2), (4), and (5)), becomes:

$$y=m_y+\left\{\left(\sqrt{\frac{1}{4}{\left(\frac{c_{xx}}{s_{xx}}-\frac{c_{yy}}{s_{yy}}\right)}^2+\frac{c_{xy}^2}{s_{xx}{s}_{yy}}}-\frac{1}{2}\left(\frac{c_{xx}}{s_{xx}}-\frac{c_{yy}}{s_{yy}}\right)\right)/\left(\frac{c_{xy}}{s_{yy}}\right)\right\}(x-m_x)$$

(18)

This is the generalized eigenproblem solution for a reversable relation between two measured variables with different errors.

In the special case of the equal variances of the errors, s_xx = s_yy, Equation (18) reduces to the following expression:

$$y=m_y+\frac{\left(c_{yy}-c_{xx}\right)+\sqrt{{\left(c_{yy}-c_{xx}\right)}^2+{4c}_{xy}^2}}{2c_{xy}}\left(x-m_x\right)$$

(19)

Equation (19) defines the orthogonal regression which is obtained from the general solution (18) in assumption of the equal precision (measured by the variances) in observations by both variables.

Another special case is defined by the proportion:

$$\frac{c_{xx}}{s_{xx}}=\frac{c_{yy}}{s_{yy}},\hspace{1em}or\hspace{1em}\frac{c_{xx}}{c_{yy}}=\frac{s_{xx}}{s_{yy}}$$

(20)

This means that the quotient of a variable’s variance by its error’s variance is equal for both variables. The same proportion holds if the ratio of the variances of two variables equals the ratio of the variances of their errors. In assumption (20), Equation (18) simplifies to:

$$y=m_y+\left(\frac{\sqrt{c_{xy}^2}}{c_{xy}}\right)\sqrt{\frac{s_{yy}}{s_{xx}}}\left(x-m_x\right)=m_y+sgn\left(c_{xy}\right)\sqrt{\frac{c_{yy}}{c_{xx}}}\left(x-m_x\right)$$

(21)

The covariance and correlation are of the same sign, and the standard deviation equals the square root of the variance, ${\sigma }_x=\sqrt{c_{xx}}$ (similarly to the other variable); therefore, the result (21) coincides with Equation (5) for diagonal regression. The same results (19)–(21) can be obtained by the maximum likelihood method based on the normal distribution of errors [9,10], but the considered approach dealing with quadratic forms (6)–(11) does not depend on a specific distribution of the errors.

Another reason for choosing the diagonal regression when the errors of observation are unknown could be as follows: let us estimate the error’s variance for each variable using the residual variance of this variable by the other available predictor. In the case of two variables, the residual variance of the model (1)–(2) equals the variance of the dependent variable, multiplied by the term 1 − r², where r is the coefficient of pair correlation, and a similar construct holds for the second model (3)–(4):

$$s_{yy}=c_{yy}\left(1-r^2\right),\hspace{1em}{s}_{xx}=c_{xx}\left(1-r^2\right)$$

(22)

From (22), the first proportion (20) can be derived; thus, in assumption of the residual variances serving as estimates of the unobserved error variances, the diagonal regression (21) can be obtained.

3. Modeling for Many Variables

In a general case of many variables, an implicit function of their connection can be presented in the series by variables in the linear, quadratic, and other order of power and the mix effects. All these items can be considered as new additional variables, and the function can be written as a general linear form:

$${\alpha }_0+{\alpha }_1{z}_{1i}+{\alpha }_2{z}_{2i}+\dots +{\alpha }_n{z}_{ni}=0$$

(23)

where $z_{ji}$ denotes each jth theoretical unobserved variable without the errors (j = 1, 2, …, n—the total number of variables) in the ith observation, which is an extension of the equation (6) from two to many variables. The observed variables in each ith point (i = 1, 2, …, N—the total number of observations in the sample) can be defined as:

$$x_{ij}=z_{ji}+{\epsilon }_{ij}$$

(24)

in which ${\epsilon }_{ij}$ is an error in the ith observation by jth variable, and the errors by each variable are assumed to have zero mean value. Substituting (24) into (23), and taking into account the identical relation (23), yields:

$${\alpha }_0+{\alpha }_1{x}_{i1}+\dots +{\alpha }_n{x}_{in}={\alpha }_1{\epsilon }_{i1}+\dots +{\alpha }_n{\epsilon }_{in}$$

(25)

Suppose only one variable is observed with errors, for example, the first variable: ${\epsilon }_{i1}\ne 0$, but all other errors equal zero. Dividing by the first alpha coefficient ${\alpha }_1$ reduces (25) to the explicit model for x₁ by new parameters a:

$$x_{i1}=\left(-\frac{{\alpha }_0}{{\alpha }_1}\right)+\left(-\frac{{\alpha }_2}{{\alpha }_1}\right)x_{i2}+\dots +\left(-\frac{{\alpha }_n}{{\alpha }_1}\right)x_{in}+{\epsilon }_{i1}\equiv a_0+a_2{x}_{i2}+\dots +a_n{x}_{in}+{\epsilon }_{i1}$$

(26)

Then, we can build the multiple linear OLS regression of outcome x₁ by the other variables. The solution for this model (26) with the centered variables can be presented as the inverted covariance matrix of n − 1 predictors multiplied by the vector of their covariances with the outcome variable x₁; the intercept can be found by this equation in the point of mean values of the variables. Instead of the first variable, assumption on the error in another variable leads to its own regression by the remaining n-1 variables.

All possible n regressions of each variable by the other predictors can be obtained in one matrix inversion of all n variables. Consider the matrix X of the order N-by-n with elements x_ij of ith observations by all jth variables. Suppose the variables are centered, so there are no intercepts in the models. Then, the matrix C = X′X is the sample variance–covariance matrix of the nth order (a generalization of the matrix in (8)). Let C⁻¹ be the inverted matrix, with its elements denoted by the upper indices, (C⁻¹)_jk = c^jk (these are the cofactors of the elements of the covariance matrix), and the diagonal elements presented in the diagonal matrix D:

$$D=diag\left(C^{-1}\right)$$

(27)

As is known, matrix C⁻¹ contains parameters of all n regressions of each variable by all the others. More specifically, any of the models of one variable by the rest of them can be written in the so-called Yule’s notations as follows:

$$x_j=a_{j0}+a_{j1.}{x}_1+a_{j2.}{x}_2+\dots +a_{j,j-1.}{x}_{j-1}+a_{j,j+1.}{x}_{j+1}\dots ...+a_{jn.}{x}_n+{\epsilon }_j$$

(28)

where a_jk. denotes the coefficient of the jth regression by the kth variable, and the dot denotes the rest of the variables [9,40]. The non-diagonal elements in any jth row of matrix C⁻¹ divided by the diagonal element in the same row and taken with opposite signs coincide with coefficients of regression x_j by all other xs (28) that can be presented as the following matrix of all parameters of all the models:

$$A\equiv D^{-1}{C}^{-1}=\left(\begin{array}{c}{c}^{11}/c^{11}\hspace{1em}{c}^{12}/c^{11}\hspace{1em}\dots \hspace{1em}{c}^{1n}/c^{11}\\ c^{21}/c^{22}\hspace{1em}{c}^{22}/c^{22}\hspace{1em}\dots \hspace{1em}{c}^{2n}/c^{22}\\ \hspace{1em}\dots \hspace{1em}\dots \hspace{1em}\dots \hspace{1em}\dots \hspace{1em}\dots \hspace{1em}\dots \\ c^{n1}/c^{nn}\hspace{1em}{c}^{n2}/c^{nn}\hspace{1em}\dots \hspace{1em}{c}^{nn}/c^{nn}\end{array}\right)=\left(\begin{array}{l} 1\hspace{1em}\hspace{0.33em}\hspace{0.33em}-a_{12.}\hspace{1em}\hspace{0.33em}\dots \hspace{0.33em}\hspace{1em}-a_{1n.}\\ -a_{21.}\hspace{1em}\hspace{0.33em}\hspace{0.33em}1\hspace{1em}\hspace{0.33em}\dots \hspace{0.33em}\hspace{1em}-a_{2n.}\\ \dots \hspace{1em}\dots \hspace{1em}\dots \hspace{1em}\dots \hspace{1em}\dots \hspace{1em}\dots \\ -a_{n1.}\hspace{1em}-a_{n2.}\hspace{1em}\hspace{0.33em}\dots \hspace{0.33em}\hspace{1em}1\end{array}\right)$$

(29)

Additionally, the product Am of matrix A (29) by the vector of mean values m of all predictors coincides with the vector of intercepts a₀ for all regressions (28):

$$a_{j0}=m_j-(a_{j1.}{m}_1+a_{j2.}{m}_2+...+a_{j,j-1.}{m}_{j-1}+a_{j,j+1.}{m}_{j+1}+...+a_{jn.}{m}_n)$$

(30)

The product of the data matrix X by the transposed matrix (29) yields the total matrix E of N-by-n order consisting of the residual errors ${\epsilon }_{ij}$ in all ith observations of all jth regression models (28):

$$E=X{A}^{\prime }=X{C}^{-1}{D}^{-1}$$

(31)

Then, the matrix of the errors’ variance–covariances can be found by the expression:

$$S=E\prime E=D^{-1}{C}^{-1}X\prime X{C}^{-1}{D}^{-1}=D^{-1}{C}^{-1}{D}^{-1}$$

(32)

because $X^{\prime }X{C}^{-1}=I$ is the identity matrix. In assumption of errors’ independence by different outcome variables, matrix (32) can be reduced to its diagonal:

$$S=diag\left(S\right)=diag\left(D^{-1}{C}^{-1}{D}^{-1}\right)=D^{-1}.$$

(33)

Notably, the values (27) are the so-called variance inflation factors (VIFs), and their reciprocal values (33) are the residual sums of squares in the regressions of each variable x_j by the rest of them:

$$S_{jj}=D_{jj}^{-1}={VIF}_j^{-1}=c_{jj}\left(1-R_{j.}^2\right)$$

(34)

where $R_{j.}^2$ is the coefficient of multiple determination in the models of each x_j by all the other variables. In the case n = 2, Equation (34) reduces to the expression (22), because for two variables, the coefficients of determination coincide with the squared pair correlation, $R_{1.}^2=R_{2.}^2=r^2$.

Similar to obtaining the geometric mean pair regression (5) from the slopes of the two regressions (2) and (4), if expressing one variable, x_j, from the regressions for all other predictors (28) using the coefficients (29), it is easy to see that all these models would coincide if the following relations for the coefficient (29) hold by all the variables:

$$a_{jk.}\cdot \frac{1}{a_{kj.}}=\frac{c^{jk}}{c^{jj}}\frac{c^{kk}}{c^{kj}}=\frac{c^{kk}}{c^{jj}}\equiv {\left({\tilde{a}}_{jk.}\right)}^2$$

(35)

where the transposed elements of the symmetric inverted matrix are equal, $c^{jk}=c^{kj}$, and the coefficient with a tilde denotes the built parameter of the geometric mean regression model. These parameters can be defined from (35) as ${\tilde{a}}_{jk.}={(c^{kk}/c^{jj})}^{1/2}$. Using them in Equation (28) for the centered variables yields:

$$x_j=\sqrt{\frac{c^{11}}{c^{jj}}}sgn\left(c^{j1}\right)x_1+\sqrt{\frac{c^{22}}{c^{jj}}}sgn\left(c^{j2}\right)x_2+...+\sqrt{\frac{c^{nn}}{c^{jj}}}sgn\left(c^{jn}\right)x_n$$

(36)

This expression presents a heuristic generalization of the geometric mean (5) or diagonal regression (21) for a case of many variables. Recalling the Cramer’s rule that a diagonal element c^jj of the inverted matrix equals the quotient of the first minor (which is a determinant of the matrix C_(-j) without the jth row and column, equal to the cofactor for the diagonal elements) and the determinant of the whole matrix, we can write the relation:

$$c^{jj}=\det \left(C_{\left(-j\right)}\right)/\det \left(C\right)$$

(37)

Substituting this equation into (36) and canceling the total determinant, we can rewrite this equation in a more symmetric form:

$$\sqrt{\det \left(C_{\left(-j\right)}\right)}{x}_j=\sqrt{\det \left(C_{\left(-1\right)}\right)}sgn\left(c^{j1}\right)x_1\dots +\sqrt{\det \left(C_{\left(-n\right)}\right)}sgn\left(c^{jn}\right)x_n$$

(38)

This expression has an interesting interpretation in terms of the generalized variance, defined as the determinant of the covariance matrix [41]: a value of each jth coefficient of the diagonal regression (38) equals the square root of the generalized variance of all additional variables to the x_j variable.

Let us return to the general relation (25) and rewrite it for the centered variables in the matrix form as $X\alpha =E\alpha$, where E is the matrix of the unknown errors by all observations in the model (25). The scalar products of these vectors by themselves yield the expression:

$${\alpha }^{\prime }{X}^{\prime }X\alpha ={\alpha }^{\prime }{E}^{\prime }E\alpha$$

(39)

where the matrices C = X′X and S = E′E are the nth order extensions of the second-order matrices in (8) and (9). For solving this problem for vector $\alpha$ of n parameters, we can find the minimum of the quadratic form on the left-hand side of (39) subject to the normalizing condition on its right-hand side (needed for the identifiability of the parameters defined up to an arbitrary constant). It corresponds to presenting (39) as the Rayleigh quotient of two quadratic forms, ${\alpha }^{\prime }C\alpha /\alpha \prime S\alpha$, reduced to the conditional least squares problem (10), and then to the generalized eigenproblem (11) for the nth order covariance matrices C and S of the variables and their errors, respectively.

For unknown errors of observations, let us estimate the error’s variance for each variable as the residual variance of this variable by all the rest of predictors, as it is considered in the case of two variables in the relation (22). For the case of independent errors, we substitute the diagonal matrix of variances (33) into the generalized eigenproblem (11), which produces the following relation:

$$C\alpha =\lambda D^{-1}\alpha$$

(40)

For the more general case of the possibly correlated errors, we substitute the variance–covariance matrix (32) into the generalized eigenproblem (11) which yields the expression:

$$C\alpha =\mu D^{-1}{C}^{-1}{D}^{-1}\alpha$$

(41)

where the eigenvalue is renamed to $\mu$ to distinguish it from eigenvalue $\lambda$ in the problem (40). The matrix D is defined by the inverted matrix C⁻¹ (27); therefore, both last generalized eigenproblems can be expressed via the variance–covariance matrix of the variables. Transforming (40) to the regular eigenproblem yields the equation:

$$\left(DC\right)\alpha =\lambda \alpha$$

(42)

and a similar transformation of (41) leads to the equation:

$$({DC)}^2\alpha =\mu \alpha$$

(43)

If the matrix DC is multiplied by the problem (42), we derive the relation:

$$\left(DC\right)\left(DC\right)\alpha =\lambda \left(DC\right)\alpha ={\lambda }^2\alpha$$

(44)

Therefore, the eigenvalues in (43) equal the squared eigenvalues in (42), ${\lambda }^2=\mu$. However, the sets of eigenvectors of both problems (42) and (43) coincide; thus, the eigenvector solution related to the minimum eigenvalue in any of these two problems is the same. It is an unexpected result that this solution does not depend on the assumption of the correlated or non-correlated errors used in (40) and (41), respectively. Therefore, it is sufficient to solve the simpler problem (42) which, utilizing the definition (27), can be presented as follows:

$$diag\left(C^{-1}\right)C\alpha =\lambda \alpha$$

(45)

The solution (45) for the eigenvector related to the minimum eigenvalue ${\lambda }_{min}$ defines the functional implicit multiple linear regression (23).

Suppose all diagonal elements of the matrix C⁻¹, or the errors’ variances are the same; thus, diag(C⁻¹) is a scalar matrix, and the problem (45) is essentially the eigenproblem of a covariance matrix. Then, its maximum ${\lambda }_{max}$ and several next big eigenvalues define the main eigenvectors used in the well-known principal component analysis (PCA), and in the regression modeling of the PCA scores employed in place of the original predictors for a multicollinearity reduction. On the other hand, ${\lambda }_{min}$ of the covariance, or correlation matrix, defines the eigenvector which presents the coefficients of the orthogonal regression [5]. Although they are based on the same eigenproblem, the PCA and regression on PCA differ from the orthogonal regression, as well as from the functional implicit regression in the case of different variances of errors.

Problem (45) presents a generalization of the solution (16)–(18) to the multiple models with n > 2 variables. Such a model is symmetrical by all variables, and any one of them can be expressed via the others from this unique equation. Eigenproblem (45) with n variables reduces to the explicit eigenproblem (12) if n = 2. The errors’ variances s_xx and s_yy in (12) are extended to many variables with the errors’ variances defined by (34) in a general case of n variables.

The eigenproblem (45) of the non-symmetric matrix can be transformed to the eigenproblem of a symmetric matrix which is more convenient in numerical calculations because it produces a higher precision of estimations. For this aim, let us rewrite problem (42) as the following expression:

$${(D}^{1/2}C{D}^{1/2})\left(D^{-1/2}\alpha \right)=\lambda \left(D^{-1/2}\alpha \right)$$

(46)

where D^1/2 is the square root of the diagonal matrix D (27). The problem (46) can be represented as the following eigenproblem:

$$B\beta =\lambda \beta$$

(47)

where the symmetric matrix, B, and its eigenvectors are defined by the relations:

$${B=D}^{1/2}C{D}^{1/2},\hspace{1em}\hspace{1em}\beta =D^{-1/2}\alpha$$

(48)

The regular eigenproblem of the symmetric matrix (47) can be solved by most modern statistical and mathematical software packages. When the eigenvector $\beta$ for the minimum eigenvalue in (47) is found, the original eigenvector $\alpha$ can be obtained from (48) by the inversed transformation:

$$\alpha =D^{1/2}\beta$$

(49)

Taking into account the relations (27) and (37), we can find each jth element of the diagonal matrix in (49):

$${\left(D^{1/2}\right)}_j=\sqrt{c^{jj}}=\sqrt{\frac{det\left(C_{\left(-j\right)}\right)}{\det \left(C\right)}}$$

(50)

Then, the original estimated model (25) with parameter alpha (49) can be presented for the centered variables as follows:

$$\sqrt{det\left(C_{\left(-1\right)}\right)}{\beta }_1{x}_1+\sqrt{det\left(C_{\left(-2\right)}\right)}{\beta }_2{x}_2+\dots +\sqrt{det\left(C_{\left(-n\right)}\right)}{\beta }_n{x}_n=0$$

(51)

where the common constant of determinant det(C) (50) is cancelled. In the case of an ill-conditioned or even singular sample covariance matrix C, the inverted matrix could not exist, but it is still possible to present the solution (51) via the determinants without each individual variable.

The functional implicit linear regression (51) obtained with the errors assumed by all variables can be solved with respect to any one variable x_j, similar to the heuristically constructed diagonal regression (38). Both models (38) and (51) have parameters proportional to the square root of the generalized variances; however, in place of the sign functions in (38) there are coefficients $\beta$ (51) of the eigenproblem (47). Finding one of two possible signs for each variable presents an additional problem in building the model (38), while the coefficients $\beta$ are uniquely defined in solving the eigenproblem (47) for the minimum eigenvalue ${\lambda }_{min}$.

For the characteristics of quality, the minimum eigenvalue ${\lambda }_{min}$ of the standardized matrix B (48) can serve as a measure of the residual sum of squares $S_{resid}^2$, taken by the shortest distances from the observations to the hyperplane of the orthogonal regression in the metric of the matrix D^1/2. An analogue of the coefficient of determination R² in multiple regression can also be constructed as one minus the quotient of the minimum eigenvalue to the mean of all eigenvalues:

$${S_{resid}^2=\lambda }_{min},\hspace{1em}\hspace{1em}{R}^2=1-\frac{{\lambda }_{min}}{\frac{1}{n}{\sum }_{j=1}^n{\lambda }_j}$$

(52)

4. Numerical Examples

For the first example, consider the simulated data for the conic section curve defined by the implicit equation of the second order:

$$x^2+2xy+y^2+2x+10y=0$$

(53)

This corresponds to the parabola with its axis of symmetry extending at the angle −45° to the x-axis. Solving this quadratic equation with respect to the y variable yields two branches of this parabola,

$$y_1=-\left(x+5\right)+\sqrt{8x+25},\hspace{1em}{y}_2=-\left(x+5\right)-\sqrt{8x+25}$$

(54)

which are shown in Figure 1.

Having a set of numeric data on x and y variables, it is still a problem to restore such a parabola equation because it has not the unique, but dual values of y at each x, so a regression of y by x could produce a line of the axis of symmetry between the branches. By the same reason, building a quadratic regression of x by y also has a problem of dual x values for the same y in the upper branch, which would distort the model. Separate modeling for the branches (54) requires splitting the data related to each branch and application of the nonlinear estimation.

If the numeric values on the x variable and on y variable defined in (54) were obtained with an additional noise of errors, it increases the difficulty of the adequate restoration of the original implicit Equation (53). The higher variability in the errors, the more fuzzified the scatter plot of the original function hidden behind the empirical data.

An example for the implicit function (53) with data by x taken from −3.1 to 6.0 with the step 0.1, for the sample size N = 92, was used to build the values of the functions (54). The random noise of normal distribution N(0, 1) with zero mean value and the standard deviation std = 1 was added to x and y variables. These values are presented in the scatter plot shown in Figure 2.

To restore the original function from the data obtained with errors, we can rewrite Equation (53) as a generalized linear implicit function:

$$a_0+a_1{u}_1+a_2{u}_2+a_3{u}_3+a_4{u}_4+a_5{u}_5=0$$

(55)

where the notations for the new variables, u_j, are:

$${u_1=x}^2,\hspace{1em}{u}_2=xy,\hspace{1em}{u}_3=y^2,\hspace{1em}{u}_4=x,\hspace{1em}{u}_5=y$$

(56)

The coefficients in (55) for the original implicit function (53) are as follows:

$$a_0=0,\hspace{1em}{a}_1=1,\hspace{1em}{a}_2=2,\hspace{1em}{a}_3=1,\hspace{1em}{a}_4=2,\hspace{1em}{a}_5=10$$

(57)

Applying the considered above approach for evaluating these parameters by the data distorted with errors, we can build each variable u_j regression by the other u-variables, find their residual variances, and use them for the errors’ estimates in the generalized eigenproblem. The obtained results are shown in

Table 1. Original function, OLS regressions, and eigenvector (parabola example).

Parameters	Original Function	Model for u1	Model for u2	Model for u3	Model for u4	Model for u5	Eigenvector
a0	0	−6.570	−4.117	5.391	0.081	1.130	0.127
a1	1	1.000	0.613	0.648	−0.109	0.021	0.096
a2	2	0.364	1.000	1.459	0.045	0.078	0.150
a3	1	0.106	0.400	1.000	0.010	0.074	0.088
a4	2	−2.005	1.390	1.115	1.000	0.070	0.111
a5	10	0.544	3.392	11.629	0.097	1.000	0.974
S2resid	0	54.907	92.495	337.042	2.974	2.134	0.380
R2	1	0.664	0.915	0.975	0.622	0.952	0.976

, the columns in which present the intercept and parameters of the original function (53), five OLS regressions (29), and the general eigenvector solution (51).

The last two rows in Table 1 show the residual variance S²_resid and the coefficient of multiple determination R² for each OLS regression, as well as their analogues for the eigenvector solution (52). For the original exact function (53), S²_resid = 0, and R² = 1; therefore, by comparison, it can be determined which models are of a better quality than the others.

Table 2. The models normalized by the first coefficients (parabola example).

	Original Function	Model for u1	Model for u2	Model for u3	Model for u4	Model for u5	Eigenvector
a0	0	−6.570	−6.719	8.315	−0.747	53.435	1.318
a1	1	1.000	1.000	1.000	1.000	1.000	1.000
a2	2	0.364	1.632	2.250	−0.412	3.701	1.554
a3	1	0.106	0.653	1.543	−0.091	3.482	0.910
a4	2	−2.005	2.268	1.720	−9.211	3.305	1.151
a5	10	0.544	5.535	17.937	−0.897	47.286	10.120

presents parameters in each model divided by the coefficient for the first variable, which is more convenient for comparison across the models. It is easy to see that some models reconstruct the original model more accurately than the other models.

To find a measure of this closeness, we calculate pair correlations between the solutions. The correlations of the vectors in the columns of Table 2 (except for the intercepts, which depend on the slope parameters) are presented in

Table 3. Correlations between the normalized solutions (parabola example).

	Original Function	Model for u1	Model for u2	Model for u3	Model for u4	Model for u5	Eigenvector
Original function	1.000	0.181	0.978	0.996	0.054	0.994	0.996
Model for u1	0.181	1.000	0.012	0.242	0.975	0.234	0.258
Model for u2	0.978	0.012	1.000	0.956	−0.132	0.953	0.958
Model for u3	0.996	0.242	0.956	1.000	0.124	1.000	0.999
Model for u4	0.054	0.975	−0.132	0.124	1.000	0.119	0.134
Model for u5	0.994	0.234	0.953	1.000	0.119	1.000	0.998
Eigenvector	0.996	0.258	0.958	0.999	0.134	0.998	1.000
Mean Correlation	0.840	0.380	0.745	0.863	0.255	0.860	0.869

.

Table 3 shows that many models are highly correlated with the original function and among themselves. The last row of the mean correlations also indicates that the OLS models for u₂, u₃, and u₅ (56), as well as the eigenvector solution, are the best performers, with high correlations with the original function.

Using any obtained solution for regression parameters from Table 1, we can calculate the values of the parabola (55) and (56) branches by the formula:

$$y_{\mathrm{1,2}}=\frac{-(a_{2}x+a_5)\pm \sqrt{{(a_{2}x+a_5)}^2-4a_3(a_1{x}^2+a_{4}x+a_0)}}{2a_3}$$

(58)

For illustration, using parameters of the solution by the eigenvector in the last column of Table 1, we can restore the parabola from the data with random noise—Figure 3 presents the original parabola and the restored curve.

The results of the eigenvector solution with the data distorted by the random noise yield a good approximation for the original exact data. The difference in the restored from the original values of the function can be estimated by the mean absolute error (MAE) or by the standard deviation (STD). For the results presented in Figure 3, these estimates are MAE = 0.442 and STD = 0.505, that are acceptable values in comparison with the standard deviation used in the random normal noise N(0, 1) added to x and y variables.

For the second numerical example, the dataset on physicochemical properties of the red wine [42] was taken (freely available at the site Wine Quality—UCI Machine Learning Repository; also available through the link Red Wine Quality | Kaggle). From that total dataset of 1599 observations and 12 variables, the following 10 variables describing chemical features were used: x₁—fixed acidity; x₂—volatile acidity; x₃—citric acid; x₄—residual sugar; x₅—chlorides; x₆—free sulfur dioxide; x₇—total sulfur dioxide; x₈—pH; x₉—sulphates; and x₁₀—alcohol.

To predict each of these characteristics using the others, we considered the outcome variable in the regression model, so ten such models could be built. These models differ and do not reduce from one to another one. However, it was possible to assume that all these characteristics were connected among themselves, and to construct one implicit relation of their interdependence by which any variable can be expressed via the rest of them. The next three tables are organized similarly to the first three tables (with an exception of the original function which is not known in these data).

Table 4. OLS regressions and eigenvector (wine example).

	Model for x1	Model for x2	Model for x3	Model for x4	Model for x5	Model for x6	Model for x7	Model for x8	Model for x9	Model for x10	Eigen-vector
a0	−25.11	0.06	0.26	1.42	−0.42	24.88	−213.19	−3.59	−0.19	−6.08	−1.09
a1	1.00	−0.03	−0.06	−0.10	0.01	−0.52	6.09	0.05	−0.01	0.11	0.04
a2	−1.83	1.00	0.44	−0.66	−0.08	9.91	−37.49	−0.09	0.20	0.19	−0.27
a3	−5.03	0.66	1.00	−0.89	−0.09	11.15	−53.99	0.03	−0.04	−1.85	−0.38
a4	−0.06	−0.01	−0.01	1.00	0.00	−0.53	−1.72	0.00	0.01	−0.07	0.00
a5	6.74	−0.97	−0.75	−1.95	1.00	−9.83	76.88	0.58	−1.46	6.03	0.85
a6	−0.01	0.00	0.00	−0.02	0.00	1.00	−1.97	0.00	0.00	−0.01	0.00
a7	0.01	0.00	0.00	−0.01	0.00	−0.22	1.00	0.00	0.00	0.01	0.00
a8	5.19	−0.16	0.04	−0.23	0.08	−7.89	39.37	1.00	−0.02	−1.48	0.23
a9	−0.49	0.17	−0.02	0.63	−0.10	−1.74	−8.30	−0.01	1.00	−1.02	−0.11
a10	0.13	0.00	−0.03	−0.14	0.01	−0.47	5.23	−0.02	−0.03	1.00	0.02
S2resid	1.02	0.02	0.01	1.81	0.00	56.15	497.92	0.01	0.02	0.87	0.38
R2	0.66	0.43	0.68	0.09	0.32	0.49	0.54	0.55	0.25	0.23	0.81

presents all ten OLS regression parameters (29) (the columns in this table correspond to the rows in the matrix on the right-hand side of (29)), as well as the general eigenvector solution (51).

Each column in Table 4 shows the intercept and the coefficients of regression which are given as in the implicit form (23); thus, to obtain the actual OLS model (28) in explicit form, we changed the signs of all coefficients. For instance, the model for the outcome x₁ is:

$$x_1=25.11+1.83x_2+5.03x_3+\dots -0.13u_{10}$$

(59)

The eigenvector model given in Table 4, is as following:

$$-1.09+0.04x_1-0.27x_2+\dots +0.02u_{10}$$

(60)

Each of the OLS regressions presents a model of the dependence of one particular variable by all the others of them; for example, the model for x₁ in Equation (59) can find the values of the dependent variable by the given values of its predictors. In contrast to the OLS models, the generalized linear relation (60) shows how, and in which proportions, all chemical components in this example can be combined in the relation of their interconnection. This can facilitate examination and interpretation of the components in their compound of a better property product.

The two bottom rows in Table 4 present the residual variance and the coefficient of multiple determination R² for each OLS regression model, as well as their analogues for the eigenvector solution (52). Most of the models were of good quality.

For the sake of comparison, we divided the parameters in each model by its coefficient for the first variable, which yielded the results presented in

Table 5. The models normalized by the first coefficients (wine example).

	Model for x1	Model for x2	Model for x3	Model for x4	Model for x5	Model for x6	Model for x7	Model for x8	Model for x9	Model for x10	Eigen-vector
a0	−25.11	−1.82	−4.27	−14.55	−41.78	−48.04	−35.03	−66.23	18.57	−52.93	−26.38
a1	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00
a2	−1.83	−30.48	−7.33	6.69	−8.07	−19.12	−6.16	−1.74	−19.40	1.69	−6.56
a3	−5.03	−20.12	−16.68	9.09	−9.40	−21.52	−8.87	0.62	3.81	−16.14	−9.11
a4	−0.06	0.20	0.10	−10.22	−0.16	1.02	−0.28	−0.02	−0.72	−0.60	−0.04
a5	6.74	29.62	12.57	19.89	100.30	18.97	12.63	10.65	141.36	52.50	20.56
a6	−0.01	−0.10	−0.04	0.17	−0.03	−1.93	−0.32	−0.03	0.06	−0.06	−0.09
a7	0.01	0.04	0.02	0.06	0.02	0.43	0.16	0.02	0.03	0.08	0.04
a8	5.19	4.91	−0.64	2.33	8.20	15.23	6.47	18.46	2.30	−12.87	5.61
a9	−0.49	−5.20	0.37	−6.41	−10.31	3.36	−1.36	−0.22	−96.79	−8.89	−2.59
a10	0.13	−0.12	0.43	1.46	1.04	0.91	0.86	−0.33	2.42	8.71	0.47

.

There is a wide variability in the parameters across the models in Table 5; therefore, to estimate the closeness between these solutions, we identified pair correlations.

Table 6. Correlations between the normalized solutions (wine example).

	Model for x1	Model for x2	Model for x3	Model for x4	Model for x5	Model for x6	Model for x7	Model for x8	Model for x9	Model for x10	Eigen-vector
Model for x1	1	0.85	0.85	0.32	0.75	0.93	0.98	0.79	0.59	0.63	0.93
Model for x2	0.85	1	0.88	0.25	0.79	0.91	0.93	0.55	0.65	0.69	0.93
Model for x3	0.85	0.88	1	0.13	0.72	0.88	0.9	0.36	0.49	0.76	0.88
Model for x4	0.32	0.25	0.13	1	0.71	0.07	0.33	0.38	0.76	0.63	0.49
Model for x5	0.75	0.79	0.72	0.71	1	0.63	0.81	0.52	0.89	0.92	0.93
Model for x6	0.93	0.91	0.88	0.07	0.63	1	0.95	0.69	0.41	0.51	0.87
Model for x7	0.98	0.93	0.9	0.33	0.81	0.95	1	0.71	0.65	0.71	0.97
Model for x8	0.79	0.55	0.36	0.38	0.52	0.69	0.71	1	0.45	0.21	0.65
Model for x9	0.59	0.65	0.49	0.76	0.89	0.41	0.65	0.45	1	0.82	0.78
Model for x10	0.63	0.69	0.76	0.63	0.92	0.51	0.71	0.21	0.82	1	0.83
Eigenvector	0.93	0.93	0.88	0.49	0.93	0.87	0.97	0.65	0.78	0.83	1
Mean correlation	0.76	0.74	0.68	0.41	0.77	0.68	0.8	0.53	0.65	0.67	0.82

shows the pair correlations of the vectors from the columns in Table 5.

Correlations were determined by the regressions’ coefficients only, without the intercepts. Many of the models are closely related. The bottom row of the mean correlations in the matrix columns in Table 6 indicates that the last model had the highest mean value; thus, the eigenvector solution presents a good compromise between the partial regression models of each one variable by the others of them.

5. Summary

This paper describes a statistical estimation of interconnections among the variables presented by the implicit functions. In the regular regression analysis for a given dataset, there could be many regressions with different variables taken as the dependent; however, building an implicit relation yields one unique invertible model of connection between all the variables. This kind of statistical modeling rather corresponds to restoring existing physical laws between the variables than to an approximation of the dependence of one outcome variable by the other predictors.

Modeling of the implicit linear relation is expressed as the generalized eigenproblem of the covariance matrix of the variables in the metric of the covariance matrix of their errors. Derivation was performed using relations between the quadratic forms in (6)–(11), (23)–(25), and (39)–(40), which are free from assumptions on a specific distribution of the errors, which could find wider practical applications.

For unknown errors, this study suggests performing their estimation via the residual errors of the ordinary least square regression of each variable by the other ones. In the simple case of two variables, the analytical close-form solutions are presented, including the orthogonal and diagonal, or geometric mean regressions. In the case of many variables, the generalized eigenproblem can be reduced to diagonalization of the matrix built from the covariance matrix of the variables and its inversed matrix. As shown in the derivations (40)–(45), the interesting feature of this generalized eigenvector problem is that the results are the same for both cases of the correlated and non-correlated estimated errors. The generalized eigenproblem solution holds the scale invariant property as well.

Numerical examples show that the eigenproblem solution presents a good compromise between all particularly regular regressions. The considered method of modeling with variables containing errors enables a unique equation of relations between all variables to be built that can be useful in practical regression modeling.