Multi-Output Bayesian Support Vector Regression Considering Dependent Outputs

Abstract

Multi-output regression aims to utilize the correlation between outputs to achieve information transfer between dependent outputs, thus improving the accuracy of predictive models. Although the Bayesian support vector machine (BSVR) can provide both the mean and the predicted variance distribution of the data to be labeled, which has a large potential application value, its standard form is unable to handle multiple outputs at the same time. To solve this problem, this paper proposes a multi-output Bayesian support vector machine model (MBSVR), which uses a covariance matrix to describe the relationship between outputs and outputs and outputs and inputs simultaneously by introducing a semiparametric latent factor model (SLFM) in BSVR, realizing knowledge transfer between outputs and improving the accuracy of the model. MBSVR integrates and optimizes the parameters in BSVR and those in SLFM through Bayesian derivation to effectively deal with the multi-output problem on the basis of inheriting the advantages of BSVR. The effectiveness of the method is verified using two function cases and four high-dimensional real-world data with multi-output.

1. Introduction

Regression models, also known as response models, can accurately predict the output of other features by establishing a mapping relationship between data features and outputs [1]. The commonly used regression models are single-output models, i.e., models with one or more inputs but only one output. However, in real engineering problems, there are often multiple outputs. For example, in novel battery material discovery, simultaneous and comprehensive prediction of the multidimensional properties of battery electrode materials is needed to help accelerate material discovery and design [2]. In environmental forecasting, there is a need to simultaneously predict particulate matter concentrations at different air quality monitoring stations, which often have potentially nonlinear spatial correlations. Reliable and accurate predictions help in crisis response and can reduce health risks [3]. Ultra-high-performance fiber-reinforced concrete (UHPFRC) is used in a variety of civil engineering applications, and its structural behavior is closer to that of steel. To investigate the effect of component dosage on its strain and energy absorption capacity under peak tension and to optimize the material dosage, both outputs need to be predicted simultaneously [4]. Multiple outputs can be processed separately, but this method ignores the potential correlation between the outputs and results in information loss. Therefore, the correlation between the outputs can be used to build a multi-output model, which is also known as a multi-response or multi-task model.

Support vector machine (SVR) was first proposed by Vapnik based on the principle of structural risk minimization [5]. SVR uses quadratic programming to obtain predictions for a single output. Compared to other models, SVR has superior performance due to its structural risk minimization principle, which allows it to avoid overfitting and achieve better output approximation [6,7]. The Bayesian support vector machine (BSVR) introduces Gaussian process assumptions and Bayesian inference on the basis of SVR to obtain the predicted values and their distributions. The BSVR model not only obtains an estimate of the unknown sample points but also has the advantages of adaptivity and prediction error distributions of Bayesian methods [8]. Meanwhile, SVR has shown superior performance in dealing with nonlinear problems and avoiding overfitting with good generalization ability [9]. Therefore, Bayesian support vector machines have received a lot of attention in the past time, such as [9,10,11] and references therein.

Multiple output regression aims to establish a mapping from multivariate inputs to multiple outputs [12]. Despite the potential utility of BSVR, its standard form cannot handle multiple outputs. The simplest way to deal with the multiple-output problem is to model multiple outputs individually. For each output indicator, a model can be built independently. This treatment is simpler but does not take into account the correlation between outputs and is suitable for scenarios where no correlation exists between outputs [13]. Another method is chain modeling [14]. This method predicts an output, followed by predicting the next output using the predicted output as input, and so on. However, the use of chain modeling requires determining the order of the outputs and the dependencies between them.

Considering that multiple outputs are correlated and being modeled individually can lead to information loss, more and more multi-output modeling approaches have been proposed. Multi-output modeling takes advantage of the correlation between outputs so that a single output can utilize information from other outputs to obtain more accurate predictions [13]. Methods have been developed to extend the support vector machine model so that it can handle multiple outputs simultaneously. Pérez-Cruz et al. [14] transformed the pipeline in a pipeline-based model $\epsilon -SVR$ into a hyper-square pipeline by equalizing the output values of the data points located outside the pipeline. This hyperspherical insensitive zone is designed to be more effective than modeling it individually. Zhang et al. [15] proposed an extended LSSVR (ELS-SVR), which extends the original feature space using vector virtualization to represent the multi-output case as an equivalent single-output case in the extended feature space and solved using a least-squares support vector machine. Inspired by multi-task learning, Xu et al. [16] changed the weight vector of a least-squares support vector machine from one to two. One carries generic information and the other carries specific information, thus characterizing the correlation between the two outputs, which is referred to as multi-output LS-SVR (MLS-SVR). Literature [17] gives an overview of the correlation methods and also analyzes the disadvantages of the above methods: the hyperspherical ϵ-tube of M-SVR does not exhibit an advantage over a hypercubic one, ELSSVR cannot handle the negative correlations, MLSSVR does not handle well the case of only partial correlations, and the above methods do not consistently outperform single-output support vector machines.

The above methods distribute modifications to $\epsilon -SVR$ and LSSVR so that they can solve the multi-output problem. Among them, the support vectors in $\epsilon -SVR$ are sparse and only some of the samples are involved in the model construction. LSSVR transforms convex quadratic optimization problems into linear systems of equations problems, in which all the samples are involved in the model construction. These methods better utilize the correlation between the outputs and improve the model accuracy to some extent. However, these methods cannot obtain a prediction distribution similar to BSVR, which can quantify uncertainty and has good application prospects. In addition, BSVR is based on Bayesian theory, which can systematically and effectively infer the optimal hyperparameters [8]. In terms of describing the correlation of multiple outputs, the method based on $\epsilon -SVR$ does not have an accurate structure to describe the correlation of outputs. ELS-SVR only describes the correlation through a parameter greater than 0, so it cannot describe the negative correlation. MLS-SVR describes the shared information through the disassembly of the weight vector. However, as the weight describes the correlation of multiple outputs in a unified manner, it cannot describe the partial correlation of the outputs.

Therefore, this paper introduces the multi-output Gaussian process assumption based on the Bayesian support vector machine model (BSVR) while considering the variability of multiple outputs in terms of SVR trade-off parameters. A Bayesian framework is used to systematically and comprehensively optimize the original BSVR hyperparameters and the hyperparameters of the Gaussian process, which in turn provides the predicted values and probability distributions of multiple outputs. The difference between the multi-output Bayesian support vector machine (MBSVR) and single-output Bayesian support vector machine (BSVR) mainly lies in the kernel function. MBSVR uses the semiparametric latent factor model in the new kernel function, which describes the relationship between the inputs and outputs, and the outputs and outputs at the same time through the linear combinations of implicit functions so that information between them can be transferred to improve the accuracy of the model. The main contributions of our work are as listed:

• The method inherits the advantages of support vector machines in nonlinear, high-dimensional problems by introducing Bayesian derivation in support vector machines.
• Compared to other SVR-based multi-output regression methods. Based on Bayesian theory, the predicted mean and its probability distribution (uncertainty) can be obtained, and the hyperparameter optimization can be performed systematically and effectively.
• Compared with BSVR, the method combines the SLFM structure with BSVR for comprehensive optimization of parameters through information transfer between outputs and uses the shared information to improve model accuracy.
• The use of a trade-off parameter makes the method sensitive to outliers and allows for more robust performance on real datasets than multi-output Gaussian process.

The rest of the paper is structured as follows: in Section 2, the Bayesian support vector machine model and the semiparametric latent factor model are introduced. In Section 3, the new multi-output Bayesian support vector machine model is introduced. In Section 4, the model evaluation is carried out using function arithmetic and real datasets, and Section 5 concludes.

2. Related Description and Basic Theories

2.1. Single-Output Bayesian Support Vector Machine

Single-output Bayesian support vector machine (BSVR) introduces Gaussian process assumptions based on SVR to optimize the hyperparameters by Bayesian derivation [8]. BSVR is widely used because it can obtain the mean and the prediction variance of the prediction through rigorous derivation to measure the uncertainty of prediction while maintaining the advantages of SVR. In BSVR, the mapping relationship between outputs and factors can be expressed as [8]:

$$y_j=g(x_j)+{\delta }_j$$

(1)

where ${\delta }_j$ is an independently and identically distributed random error. $g(x_j)$ is the support vector machine regression model. It is a zero-mean Gaussian stochastic process whose covariance between the two outputs of different x can be expressed as:

$$k(\tilde{g}(x),\tilde{g}(x\prime )=k(x,x\prime )={\displaystyle \prod _{j=1}^n\exp (-{\theta }_j{(x_j-x_j^{\prime })}^2)}$$

(2)

where $\theta =({\theta }_1,{\theta }_2\cdots {\theta }_n)$ is the hyperparameter to be adjusted and the covariance between the two outputs is equal to the value of the corresponding kernel function. Under the assumptions of the Gaussian process, the a priori of the outputs $G={\{\tilde{g}(x_1),\tilde{g}(x_2),\cdots ,\tilde{g}(x_N)\}}^T$ can be described as:

$$p(G|\gamma )={\displaystyle \frac{1}{Z_G}}\exp (-{\displaystyle \frac{1}{2}}{(G-b)}^T{K}^{-1}(G-b))$$

(3)

$$Z_G={(2\pi )}^{N/2}|K{|}^{1/2}$$

(4)

where $K\in {ℝ}^{N\times N}$ is the input covariance matrix in the sample where training was performed, $b=[b,b,\cdots ,b]\in {ℝ}^N$. $\gamma$ denotes all hyperparameters. In this problem, the hyperparameters include the hyperparameters $\theta =({\theta }_1,{\theta }_2\cdots {\theta }_n)$ in the kernel function and the trade-off parameters $C$. Since the noise is assumed to be an independent and identically distributed random variable, the likelihood function of the sample output for the training set of samples can be expressed as [8]:

$$p(Y|G,\gamma )={\displaystyle \prod _{j=1}^{N}p(y_j-g(x_j)|G,\gamma )={\displaystyle \prod _{j=1}^{N}p({\delta }_j)}}$$

(5)

where $p({\delta }_j)$ is the probability distribution of ${\delta }_j$ with the expression:

$$p(\delta )={\displaystyle \frac{1}{Z_{\delta }}}\exp (-Cl(\delta ))$$

(6)

$$Z_{\delta }={\displaystyle \int \exp (-Cl(\delta ))d\delta }=\sqrt{2\pi /C}$$

(7)

where $l(\delta )$ is the loss function of the model and $C$ is a trade-off constant. According to Bayesian theory, the posterior distribution is obtained by synthesizing the information of the existing samples with the prior distribution. In BSVR, the likelihood function characterizes the training set sample information, while $p(G|\gamma )$ is the prior distribution of the samples. Since the prior distribution satisfies the Gaussian process assumption, it can be represented by Equation (3). In summary, the posterior distribution of satisfies [18].

$$p(G|Y,\gamma )={\displaystyle \frac{p(Y|G,\gamma )p(G|\gamma )}{p(Y|\gamma )}}$$

(8)

$p(Y|\gamma )$ is a normalization constant. Bringing (3) and (5) into (8) yields:

$$p(G|Y,\gamma )={\displaystyle \frac{1}{Z}}\exp (-C{\displaystyle \sum _{i=1}^{N}l(y_i-\tilde{g}(x)})-{\displaystyle \frac{1}{2}}{(G-b)}^T{K}^{-1}(G-b))$$

(9)

$$Z={\displaystyle \int \exp (-S(G))dG}$$

(10)

$$S(G)={\displaystyle \sum _{j=1}^{N}Cl(y_j-\tilde{g}(x_j)})+{\displaystyle \frac{1}{2}}{(G-b)}^T{K}^{-1}(G-b)$$

(11)

Thus, according to the principle of the great likelihood method of solution, maximizing the posterior distribution in (11) can be equated to:

$$\underset{G}{\min }{\displaystyle \sum _{i=1}^{N}Cl(y_i-\tilde{g}(x_i)})+{\displaystyle \frac{1}{2}}{(G-b)}^T{K}^{-1}(G-b)$$

(12)

where $C$ is the equilibrium parameter. The loss function of a support vector machine can be represented in a variety of ways, and one is the squared loss function:

$$l(\delta )={\displaystyle \frac{1}{2}}{\delta }^2$$

(13)

Minimizing the squared loss function is essentially equivalent to great likelihood estimation under the assumption that the error follows a Gaussian distribution. Bringing the loss function expression into (12) yields the new objective function as:

$$\underset{G}{\min }{\displaystyle \sum _{j=1}^N\frac{C}{2}}{e}_j^2+{\displaystyle \frac{1}{2}}{(G-b)}^T{K}^{-1}(G-b)$$

(14)

where $y_j=g(x_j)+e_j$. Solving yields an estimate of $G$ as:

$$\widehat{G}=K{(K+I/C)}^{-1}Y=K\beta +b$$

(15)

where $I$ is the unit matrix, $\beta =[{\beta }_1,{\beta }_2,\cdots {\beta }_N]={(K+I/C)}^{-1}Y$.

For the output $g(x)$ to be predicted, its joint distribution with the training set satisfies:

$$\left(\begin{array}{l}g(x)\\ G\end{array}\right)\sim N\left(\left\{\begin{array}{l}0\\ 0\end{array}\right\},\left\{\begin{array}{cc}k(x,x)& k(x,X)\\ k(X,x)& k(X,X)\end{array}\right\}\right)$$

(16)

$$k(X,x)=k{(x,X)}^T={[k(x_1,x),\cdots ,k(x_N,x)]}^T$$

(17)

(17) denotes the variance between $G$ and $g(x)$, $k(X,X)=K$. The prior of $g(x)$ still obeys Gaussian distribution:

$$\begin{array}{c}p(g(x)|Y)={\displaystyle \int p(\tilde{g}(x)|Y)p(G|Y)dG}\\ =N(\mu (x),{\mathrm{\Sigma }}^2(x))\end{array}$$

(18)

$$\begin{array}{ll}\mu (x)& =k(x,X)k{(X,X)}^{-1}\widehat{G}\\ & =k(x,X){(K+I/C)}^{-1}Y\\ & ={\displaystyle \sum _{j=1}^N{\beta }_{j}k(x,x_j)}\end{array}$$

(19)

$${\mathrm{\Sigma }}^2(x)=k(x,x)-k(x,X)k{(X,X)}^{-1}k(X,x)$$

(20)

For the Bayesian support vector machine with a squared loss function, the parameters to be optimized include $\theta$ and $C$, and the optimal values of these hyperparameters are determined by the maximum posteriori probability. The specific formula derivation and calculation methods are described in [10].

2.2. Semiparametric Latent Factor Model (SLFM)

The most used multi-output covariance structure in multi-output Gaussian processes is the linear model of coregionalization (LMC) [19]. The semiparametric latent factor model (SLFM) is a special form of LMC. The model assumes that there are $Q$ shared potential Gaussian processes, and generally, the number of potential Gaussian processes is smaller than the number of outputs. SLFM represents the output as a linear combination of $Q$ Gaussian processes. Taking two outputs as an example, two Gaussian implicit functions are chosen, $N_f=Q=2$. Assuming that $u_1(x)$ and $u_2(x)$ are obtained by sampling from the two Gaussian processes respectively, where $u_1(x)\sim \mathcal{G}\mathcal{P}(0,k_1(x,x\prime ))$, $u_2(x)\sim \mathcal{G}\mathcal{P}(0,k_2(x,x\prime ))$, $g_1(x)$ and $g_2(x)$ are obtained by linearly transforming $u_1(x)$ and $u_2(x)$:

$$g_1(x)=a_{1,1}{u}_1(x)+a_{2,2}{u}_2(x)$$

(21)

$$g_2(x)=a_{2,1}{u}_1(x)+a_{2,2}{u}_2(x)$$

(22)

where $u_1(x)$ and $u_2(x)$ obey a Gaussian process distribution with different covariance functions and are independent of each other, i.e., in the case of $q\ne q\prime$, $u_q(x)\perp u_{q\prime }(x\prime )$, $\mathrm{cov}(u_q(x),u_{q\prime }(x\prime ))=0$. Since a new covariance function can be obtained by linearly combining several covariance functions. $g_1(x)$ and $g_2(x)$ can be written as:

$$g(x)=a_1{u}_1(x)+a_2{u}_2(x)$$

(23)

where $g(x)={\left[\begin{array}{cc}{g}_1(x)& g_2(x)\end{array}\right]}^{\mathrm{T}}$, $a_1={\left[\begin{array}{cc}{a}_{1,1}& a_{2,1}\end{array}\right]}^{\mathrm{T}}$, $a_2={\left[\begin{array}{cc}{a}_{1,2}& a_{2,2}\end{array}\right]}^{\mathrm{T}}$, the variances of $g(x)$ and $g(x\prime )$ can be calculated:

$$\begin{array}{ll}{k}_M(x,x\prime )& =\mathrm{cov}(g(x),g(x^{\prime }))\\ & =a_1{(a_1)}^{\mathrm{T}}\mathrm{cov}(u_1(x),u_1(x\prime ))+a_2{(a_2)}^{\mathrm{T}}\mathrm{cov}(u_2(x),u_2(x\prime ))\\ & =a_1{(a_1)}^{\mathrm{T}}{k}_1(x,x\prime )+a_2{(a_2)}^{\mathrm{T}}{k}_2(x,x\prime )\end{array}$$

(24)

Defining $B_1=a_1{(a_1)}^{\mathrm{T}}$, $B_2=a_2{(a_2)}^{\mathrm{T}}$, we can obtain

$$k_M(x,x\prime )=B_1{k}_1(x,x\prime )+B_2{k}_2(x,x\prime )$$

(25)

$$\left[\begin{array}{c}{g}_1\\ g_2\end{array}\right]=\left[\begin{array}{c}{g}_1(x_1)\\ ⋮\\ g_1(x_N)\\ g_2(x_1)\\ ⋮\\ g_2(x_N)\end{array}\right]\sim \mathcal{N}\left(\left[\begin{array}{c}0\\ 0\end{array}\right],B_1\otimes K_1+B_2\otimes K_2\right)$$

(26)

Furthermore, consider a plurality of outputs ${\{g_i(x)\}}_{i=1}^{N_f}$ in a more generalized form:

$$g_i(x)={\displaystyle \sum _{q=1}^Q{a}_{i,q}{u}_q(x)}$$

(27)

(27) can be described using a matrix as:

$$g(x)=Au(x)$$

(28)

$$g(x)={\{{\mathrm{g}}_1(x),g_2(x),\cdots g_{N_f}(x)\}}^{\mathrm{T}}$$

(29)

$$u(x)=\left[u_1(x),u_2(x),\cdots ,u_Q(x)\right]$$

(30)

$$A=\left[\begin{array}{ccc}{a}_{1,1}& \cdots & a_{1,Q}\\ ⋮& \ddots & ⋮\\ a_{N_f,1}& \cdots & a_{N_f,Q}\end{array}\right]\in {ℝ}^{N_f\times Q}$$

(31)

Similarly, since multiple Gaussian processes are independent for multiple outputs $g(x)={[g_1(x)\cdots g_{N_f}(x)]}^{\mathrm{T}}$, the covariance of multiple outputs can be expressed as:

$$k_M(x,x\prime )={\displaystyle \sum _{q=1}^Q{A}_q{A}_q^{\mathrm{T}}}{k}_q(x,x\prime )$$

(32)

$$\left[\begin{array}{c}{g}_1\\ ⋮\\ g_{N_f}\end{array}\right]=\left[\begin{array}{c}{g}_1(x_1)\\ ⋮\\ g_1(x_N)\\ g_2(x_1)\\ ⋮\\ g_2(x_N)\\ ⋮\\ g_{N_f}(x_1)\\ ⋮\\ g_{N_f}(x_N)\end{array}\right]\sim \mathcal{N}\left(\left[\begin{array}{c}0\\ ⋮\\ 0\end{array}\right],{\displaystyle \sum _{q=1}^Q{A}_q{A}_q^{\mathrm{T}}\otimes K_q}\right)$$

(33)

where in $A_q\in {ℝ}^{N_f\times N_f}$, the elements corresponding to the $i$ output is $A_{ii\prime }^q=a_{i,q}{a}_{i\prime ,q}$. To further characterize the multi-output Gaussian process, it is necessary to determine the number of implicit functions. It has been found that the larger $Q$ is, the more flexible the model is and the more variability can be described. Some scholars have determined $Q$ to be two or the number of outputs. The increase of $Q$ will also bring about a further increase in computational cost. In order to balance the flexibility and accuracy of the model and the computational overhead, this paper will take the value of $Q$ as $N_f$. Then, the correlation can be further expressed as:

$$k_M(x,x\prime )=Rdiag[k_1(x,x\prime ),\cdots ,k_Q(x,x\prime )]R^T$$

(34)

$$A_q=r_q{r_q}^{\mathrm{T}}$$

(35)

$$R=[r_1,r_2,\cdots ,r_Q]$$

(36)

$$A=\left[\begin{array}{ccc}{a}_{1,1}& \cdots & a_{1,Q}\\ ⋮& \ddots & ⋮\\ a_{N_f,1}& \cdots & a_{N_f,Q}\end{array}\right]=R{R}^{\mathrm{T}}$$

(37)

and $Q=N_f$, $k_q(x,x\prime )={\displaystyle \prod _{d=1}^D\exp (-{\theta }_d{(x_d-x_d^{\prime })}^2)}$. The kernel function of a Gaussian process has a set of parameters $\theta \in {ℝ}^D$ to be optimized, $D$ is the sample dimension, then for each of the $Q$ implicit functions, there is a set of $\theta$, which is used to measure the importance of the inputs as equivalent to the specified outputs. Then the parameters of the kernel function to be optimized include ${\theta }_M=[{\theta }_1,{\theta }_2,\cdots ,{\theta }_{N_f}]\in {ℝ}^{N_f\times N}$. ${\sum }_0=A{A}^T\in {ℝ}^{N_f\times N_f}$ is used to describe the covariance between multiple outputs. $A$ is the upper triangular matrix and also a set of unknown hyperparameters to be optimized.

3. Multi-Output Bayesian Support Vector Regression Model

The structure of the MBSVR model is shown in Figure 1, where the left side is the SLFM structure and $g(x)$ combines linear combination of $Q$ implicit functions, according to which the variance between $g(x)$ can be quantitatively described. The right side represents the trade-off parameters in the support vector machine; for each output, there is a corresponding trade-off parameter, which is used to trade off the complexity and error of the model. The $ith$ output can be expressed as (27), where $a_{i,q}$ is the parameter to be optimized, $u_q(x)$ is Gaussian process implicit function. Based on the expression of $g_i(x)$, the model can be expressed as:

$$Y_i=g(x_i)+{\mathbf{\delta }}_i$$

(38)

where $x_j$ is the $jth$ sample and $\delta$ is an independently and identically distributed random error. C is a number greater than 0, which determines the degree of tolerance for error in the model. When C is large, the model will not allow for errors, the complexity is high, and it may be overfitted with poor generalization ability. When C is small, the model does not focus on the presence of errors, the model is simpler, and it is easy to be underfitted.

MSVR combines the SLFM structure with the support vector machine model through the Bayesian assumptions. Through the Gaussian process assumption and Bayesian derivation, the correlation between the outputs is effectively delineated, and finally, the predicted mean and probability distribution of multiple outputs are obtained.

3.1. Bayesian Assumptions for MBSVR

Assume that a multi-output modeling problem consists of $N_f$ outputs and $N$ samples. Define a vector $Y$, which characterizes the outputs of the sample points and contains $N_f\times N$ elements in the vector. Multi-output Bayesian support vectors aim to approximate the $N_f$ outputs ${\{g_i(x)\}}_{1\le i\le N_f}$ simultaneously. A more accurate model is built by considering the correlation between the outputs. In a multi-output Bayesian support vector machine, for a certain $x_j\in {ℝ}^d$, the relationship between the outputs and the factors can be expressed as (38). where ${\delta }_j\in {ℝ}^{N_f}$ is an independently and identically distributed random error whose distribution form is usually unknown. $Y_j$ is the values of multiple outputs. $g(x)$ is a support vector machine, which is a multi-output Gaussian process. Since there are multiple outputs, the multiple outputs of all samples $g(x)={\{{\mathrm{g}}_1(x),g_2(x),\cdots g_{N_f}(x)\}}^{\mathrm{T}}$ are satisfied:

$$g(x)\sim \mathcal{G}\mathcal{P}(0,{\sum }_0)$$

(39)

According to the SLFM principle, ${\sum }_0=A{A}^T$ is the parameter to be optimized. $N_f$ Gaussian process outputs can be expressed as:

$${\left\{\begin{array}{c}{\tilde{g}}_1(x_1),{\tilde{g}}_1(x_2),\cdots ,{\tilde{g}}_1(x_N)\\ {\tilde{g}}_2(x_1),{\tilde{g}}_2(x_2),\cdots ,{\tilde{g}}_2(x_N)\\ ⋮\\ {\tilde{g}}_{N_f}(x_1),{\tilde{g}}_{N_f}(x_2),\cdots ,{\tilde{g}}_{N_f}(x_N)\end{array}\right\}}^{\mathrm{T}}$$

(40)

In order to make the model satisfy the Gaussian process assumptions and to facilitate the solution, the Gaussian process output is stored using a stack as: $G={\left\{{\tilde{g}}_1(x_1),{\tilde{g}}_2(x_1),\cdots ,{\tilde{g}}_{N_f}(x_1)\cdots ,{\tilde{g}}_1(x_N),{\tilde{g}}_2(x_N),\cdots ,{\tilde{g}}_{N_f}(x_N)\right\}}^{\mathrm{T}}\in {ℝ}^{N_f\times N}$. To facilitate the derivation of the formula, it is denoted as $G={\{g_1,g_2,\cdots ,g_{N_f\times N}\}}^{\mathrm{T}}$. Then, the likelihood function of $G$ can be expressed as:

$$p(G|\gamma )={\displaystyle \frac{1}{{(2\pi )}^{N_f\times N/2}{(|K_M|)}^{1/2}}}\exp [-{\displaystyle \frac{1}{2}}{(G-b)}^T{K_M}^{-1}(G-b)]$$

(41)

where $b=[b,b,\cdots ,b]\in {ℝ}^{N_{f}N}$ denotes the mean vector of the $N_f\times N$ elements and $K_M\in {ℝ}^{N_{f}N\times N_{f}N}$ denotes the covariance matrix of $Y$. $\gamma$ denotes the parameter vector to be optimized. The definition of the covariance matrix $K_M$ is the key element that distinguishes the multi-output Gaussian process from the multi-output correlation, and the description of the multi-output correlation is also included in the covariance matrix.

Since the noise is assumed to be an independent and identically distributed random variable, the likelihood function of the sample output for a given training set of samples can be expressed as:

$$p(Y|G,\gamma )={\displaystyle \prod _{r=1}^{r=N_{f}N}p(y_r-g_r|G,\gamma )={\displaystyle \prod _{r=1}^{r=N_{f}N}p({\delta }_r)}}={\displaystyle \prod _{i=1}^{i=N_f}{\displaystyle \prod _{j=1}^{j=N}p({\delta }_{ij})}}$$

(42)

where $p(\delta )$ is the probability distribution of $\delta$ and the expression is:

$$p({\delta }_i)={\displaystyle \frac{1}{Z_{{\delta }_i}}}\exp (-C_{i}l({\delta }_i))$$

(43)

$$Z_{{\delta }_i}={\displaystyle \int \exp (-C_{i}l({\delta }_i))d\delta }=\sqrt{2\pi /C_i}$$

(44)

where $l({\delta }_i),i=1,2,\cdots ,N_f$ is the loss function of the support vector machine. $C={[C_1,C_2,\cdots ,C_{N_f}]}^{\mathrm{T}}$ is the trade-off constant. For each output, there is a corresponding trade-off constant. According to Bayesian theory, the posterior distribution of $G$ satisfies:

$$p(G|Y,\gamma )={\displaystyle \frac{p(Y|G,\gamma )p(G|\gamma )}{p(Y|\gamma )}}$$

(45)

$p(Y|\gamma )$ is a normalization constant, and further, $p(G|Y,\gamma )$ can be expressed as (See Appendix A for more details):

$$p(G|Y,\gamma )={\displaystyle \frac{1}{Z}}\exp (-C_{M}l{(y-g)}^T-{\displaystyle \frac{1}{2}}{(G-b)}^T{K_M}^{-1}(G-b))$$

(46)

where $C={[C_1,C_2,\cdots ,C_{N_f}]}^{\mathrm{T}}$, $C_M=1D\otimes C$, $1D=[1,1,\cdots ,1]\in {ℝ}^{N_f}$, $\otimes$ is Kronecker products.

$$Z={\displaystyle \int \exp (-S(G))dG}$$

(47)

$$S(G)=C_{M}l{(y-g)}^{\mathrm{T}}+{\displaystyle \frac{1}{2}}{(G-b)}^T{K_M}^{-1}(G-b)$$

(48)

Therefore, maximizing the posterior distribution according to the principle of the great likelihood method of solution can be equated to:

$$\underset{G}{\min }{C}_{M}l{(y-g)}^{\mathrm{T}}+{\displaystyle \frac{1}{2}}{(G-b)}^T{K_M}^{-1}(G-b)$$

(49)

Similar to the original support vector machine, the first term of the objective function denotes the empirical risk, and the second term, which denotes the smoothness of the function, $C_M$ is an expansion of the trade-off parameters.

3.2. Model Construction for MBSVR

As with single-output Bayesian support vector machines, MBSVR still uses the squared loss function:

$$l(\delta )={\displaystyle \frac{1}{2}}{\delta }^2$$

(50)

The squared loss function actually obeys a Gaussian probability density function [20]. Bringing in the loss function expression yields the new objective function as

$$\begin{array}{l}\underset{G}{\min }{\displaystyle \frac{1}{2}}{C}_M{e}^2+{\displaystyle \frac{1}{2}}{(G-b)}^T{K_M}^{-1}(G-b)\\ s.t.y_r=g_r+e_r\end{array}$$

(51)

where $e=\{e_1,e_2,\cdots ,e_{N_{f}N}\}$. $e_r=y_r-g_r,r=1,2,\cdots ,N_{f}N$. $C_M=1D\otimes C$. The estimate of $G$ is (See Appendix B for more details):

$$\widehat{G}=K_M{(K_M+C_h)}^{-1}Y+b=K_M\beta +b$$

(52)

where $C_h$ satisfies $C_h\odot diag(C_M)=I$, $diag(C_M)$ is in diagonal form of $C_M$, $I\in {ℝ}^{N_{f}N\times N_{f}N}$ is a unit matrix, $\beta =[{\beta }_1,{\beta }_2,\cdots {\beta }_{N_f\times N}]={(K_M+C_h)}^{-1}Y$. $\odot$ is the Hadamard product, denoting the element-by-element multiplication of the matrix. For the output to be predicted $g(x)$, its joint distribution with the training set is satisfied:

$$\left(\begin{array}{l}g(x)\\ G\end{array}\right)\sim \mathcal{N}\left(\left\{\begin{array}{l}{b}_0\\ b\end{array}\right\},\left\{\begin{array}{cc}{k}_M(x,x)& k_M(x,X)\\ k_M(X,x)& k_M(X,X)\end{array}\right\}\right)$$

(53)

$$k_M(X,x)=k_M{(x,X)}^T={[k_M(x_1,x),\cdots ,k_M(x_N,x)]}^T$$

(54)

$$k_M(X,X)=\left[\begin{array}{ccc}{k}_{11}(X,X)& \cdots & k_{1N_f}(X,X)\\ ⋮& \ddots & ⋮\\ k_{N_{f}1}(X,X)& \cdots & k_{N_f{N}_f}(X,X)\end{array}\right]$$

(55)

where $k_M(X,X)=K_M$, $b_0=[b,b,\cdots ,b]\in {ℝ}^{N_f}$.The prior of $g(x)$ still obeys a multi-output Gaussian distribution:

$$p(g(x)|G)=\mathcal{N}(\mu (x),{\mathrm{\Sigma }}^2(x))$$

(56)

$$\mu (x)=b_0+k_M(x,X){(k_M(X,X)+C_h)}^{-1}(G-b)$$

(57)

$${\mathrm{\Sigma }}^2(x)=k_M(x,x)-k_M(x,X)(k_M(X,X)+C_h){)}^{-1}{k}_M(X,x)$$

(58)

where $b_0=[b,b,\cdots ,b]\in {ℝ}^{N_f}$, $k_M(x,X)\in {ℝ}^{N_f\times N_{f}N}$,the expression is:

$$k_M(x,X)=\left[\begin{array}{ccc}{k}_{11}(x,X)& \cdots & k_{1N_f}(x,X)\\ ⋮& \ddots & ⋮\\ k_{N_{f}1}(x,X)& \cdots & k_{N_f{N}_f}(x,X)\end{array}\right]$$

(59)

$$k_{ii\prime }(x,X)=\left[k_{ii\prime }(x,x_1),k_{ii\prime }(x,x_2),\cdots ,k_{ii\prime }(x,x_N)\right],i=1,2,\cdots N_f$$

(60)

$$k_M(x,x)=\left[\begin{array}{ccc}{k}_{11}(x,x)& \cdots & k_{1N_f}(x,x)\\ ⋮& \ddots & ⋮\\ k_{N_{f}1}(x,x)& \cdots & k_{N_f{N}_f}(x,x)\end{array}\right]\in {ℝ}^{N_f\times N_f}$$

(61)

The variance of the $ith$ diagonal element of ${\mathrm{\Sigma }}^2(x)$ corresponds to the variance of $ith$ output of $x$.

3.3. Optimized Solution of Parameters

In MBSVR, the parameters to be optimized include the kernel function parameter ${\theta }_M$; the trade-off parameter $C$; and the matrix $A$, which describes the correlations between outputs. For computational convenience, the specific implementation is decomposed by Cholesky $A$ into $R{R}^T$. The optimal values of these hyperparameters are determined by the maximum a posteriori probability:

$$p(\gamma )={\displaystyle \frac{p(Y|\gamma )p(\gamma )}{p(Y)}}$$

(62)

where $\gamma =\{{\theta }_M,C,R\}$, $p(\gamma )$ is the prior distribution of the hyperparameters, and $p(Y)$ is a regularization constant that in general specifies a uniform probability distribution for the hyperparameters. Therefore, its prior distribution $p(\gamma )$ is a constant. Therefore, it is only necessary to maximize $p(Y|\gamma )$ to achieve the purpose of great likelihood estimation of the parameters:

$$p(Y|\gamma )={\displaystyle \int p(Y|G,\gamma )p(G|\gamma )dG}={\displaystyle \frac{1}{Z_G{\displaystyle \prod _{i=1}^{N_f}{Z}_{{\delta }_i}^N}}}{\displaystyle \int \exp (-S(G))dG}$$

(63)

where $Z_G={(2\pi )}^{N_f\times N/2}{(|K_M|)}^{1/2}$, $Z_{{\delta }_i}={\displaystyle \int \exp (-C_{i}l({\delta }_i))d{\delta }_i}$. $S(G)$ can be expressed as

$$S(G)=S(\widehat{G})+{\displaystyle \frac{1}{2}}{(G-\widehat{G})}^T({K_M}^{-1}+diag(C_M)\odot I)(G-\widehat{G})$$

(64)

Bringing (64) into the probability distribution in (63) yields the following equation:

$$-\ln (p(Y|\gamma ))={\displaystyle \frac{1}{2}}{C}_M{e}^2+{\displaystyle \frac{1}{2}}{\beta }^T{K}_M\beta +{\displaystyle \frac{1}{2}}\ln |I+diag(C_M)\odot K_M|+N{\displaystyle \sum _{i=1}^{N_f}\ln Z_{{\delta }_i}}$$

(65)

The hyperparameters are obtained by solving according to the minimized likelihood function. The nonlinear programming problem is solved using the “fmincon” function in MATLAB2022b. Given the initial solution, iterative optimization is performed to obtain the optimal hyperparameters. In general, the method can find the global optimal solution of the objective function, and the initial value of the parameters on the optimal solution of the parameters has less influence.

4. Numerical Experiments

4.1. Performance Metrics and Experiment Settings

The single-response Bayesian support vector machine [10] and a multi-output Gaussian process model [21] are used for comparison with the multi-output Bayesian support vector machine model (MBSVR). The three methods are denoted by (independent regression) IND, (multi-output Gaussian process) MGP, and MBSVR, respectively. The lhsdesign function in MATLAB is used to generate the training set and test set, and for the model parameters, the kernel function parameter in MBSVR is initialized to 1, the trade-off parameter C is initialized to 1000, and the range of optimality search is $[0.01,100]$ and $[1,{10}^6]$. The kernel function parameter of the multi-output Gaussian process is initialized to 1. In the multi-output Bayesian support vector machine, $A_q$ is initialized to a unit matrix. For the same hyperparameters in all three models, the same initial values and optimization ranges are assigned to allow for a fairer comparison.

To measure the generalization effect of the model, the error criterion normalized root mean square error (NRMSE) is used, and its expression is:

$$NRMS{E}_i={\displaystyle \frac{\sqrt{\frac{1}{Tn}{\displaystyle \sum _{t=1}^{Tn}(y_{it}-{\tilde{y}}_{it})}}}{\max (y_i)-\min (y_i)}}$$

(66)

where $Tn$ is the size of the test set, $i$ denotes $ith$ output, $y_{it}$ is the true value of the $t$ test set for the $i$ first output, and ${\tilde{y}}_{it}$ is the corresponding predicted value. The smaller the value of the error metric, the more accurate the model is. The performance of the model is affected not only by the training set size but also by the specific sample points. Therefore, in order to ensure the diversity of the experiments and fully reflect the model performance, repeat the experiments 100 times; 100 training sets are generated with the same number of points and statistically characterize the results. In this paper, we use the Pearson correlation coefficient for correlation analysis of the output before proceeding with the model construction, which is the most used [13].

4.2. Datasets Description

In order to evaluate the modeling effect of the three models, two function cases and four real data are selected to verify the effect of the proposed method. The functions are based on the original cases with some changes so that their outputs have a certain degree of relevance, and the selected function cases include the one-dimensional case of Forrester and the two-dimensional case of Branin, and all data details are shown in

Table 1. Information about the test dataset.

Type	Name	D	${\mathit{N}}_{\mathit{f}}$	Dataset Size	Train Size	Test Size
Numerical function	Forrester	1	2		4:1:8	100
	Branin	2	2		5:3:20	5000
Real-world datasets	Energy_efficiency	8	2	768	10:5:40	500
	Polymer	10	4	61	10:5:35	20
	Broomcorn	19	3	128	30:10:90	30
	Sarcos	21	7	44,484	10:5:50	4449

.

The specific expressions of Forrester and Branin are given in (Appendix C), and the information of Energy_efficiency, Polymer, Broomcorn, and Sarcos are as follows:

Energy_efficiency [22]. This dataset contains a total of two outputs for heat load and cooling load demand for building energy efficiency. It includes eight factors, such as lighting area, roof area, and overall height. There is a total of 768 sample points in this dataset.

Polymer [16]. The polymer dataset contains ten inputs, such as temperature and feed rate, and contains four outputs of measurements. The dataset contains a total of 61 samples.

Broomcorn [22]. This dataset is a sorghum sample from the Institute of Crop Seed Resources, Chinese Academy of Agricultural Sciences, containing a total of 128 sample points. It contains 19 inputs and 3 outputs. The outputs are the protein, lysine, and starch fractional content of the sorghum samples, respectively.

Sarcos [13,23]. This dataset is a high-dimensional, large-scale dataset. This dataset is an inverse dynamics modeling problem for a 7-degree-of-freedom anthropomorphic robotic arm that has 21 inputs (7 joint positions, 7 joint velocities, and 7 joint accelerations) and corresponding moments at 7 joints as outputs. Only the modeling results for six responses are shown.

4.3. Results and Discussions

4.3.1. Numerical Functions

From Forrester’s expression, it can be found that the output f1 is a nonlinear variation of f2. As shown in Figure 2, the actual output of the two functions is computationally highly correlated, with a Pearson correlation coefficient of 0.95.

Figure 3 shows the modeling results for the two outputs of Forrester. The bottom and top of the boxplot indicate the lower and upper quartiles, respectively, while the center depression indicates the median of the metrics. The maximum length of the vertical line at the end of the boxplot is 1.5 times the interquartile spacing. A red plus sign indicates an outlier that is outside the boundaries of the vertical line. It can be seen that when the number of sample points is small (4), the three modeling methods are roughly equivalent. As the number of training sets increases, the two modeling methods MBSVR and MGP, which consider output correlation. This phenomenon may be due to the fact that when the number of samples is small, the current information cannot support the accurate solution of relevant parameters such as correlation. Moreover, MBSVR has the best results and the most significant advantage when the size of the training set is 5 or 6, after which the modeling results of MBSVR are gradually comparable to the modeling results of MGP. This may be related to the small-sample modeling capability of SVR.

Next, a two-dimensional function Branin with two outputs is used to validate method validity. Output 2 is the original Branin function, and Output 1 is a fine-tuning of the original Branin with a linear translation added, as shown in Figure 4. These two outputs are also strongly correlated, with a Pearson correlation coefficient of 0.69.

Figure 5 shows the modeling results of Branin. Overall, the modeling method considering correlation outperforms the independent modeling method, and for both Output 1 and Output 2, the accuracy of the MBSVR is higher than that of the MGP for moderate training set sizes (11, 14, 17). For training set size 20, the accuracy of the MBSVR model is roughly comparable to that of the MGP model, and even lower than that of the MGP in some sample point cases are even lower than MGP. This phenomenon, which is the modeling advantage of MBSVR becoming progressively less significant as the training set increases, is consistent with the one-dimensional arithmetic case.

4.3.2. Real-World Dataset

According to the results of the correlation analysis, the Pearson correlation coefficient of Energy_efficiency is greater than 0.97. The Pearson correlation coefficients of the four outputs of Polymer are large, as shown in Figure 6a, and there is a strong correlation between Output 1 and Output 2 as well as between Output 3 and Output 4. As for Broomcorn, the Pearson correlation coefficient of Output 1 and Output 2 is 0.39, the correlation coefficient of Output 1 and Output 3 is −0.67, and the correlation coefficient of Output 2 and Output 3 is −0.37. The results of the output correlation analysis of Sarcos are shown in Figure 6b. The correlation of Output 4 and Output 7 is the highest, and the correlation of Output 1 and Output 2 is the lowest. Output 6 has low correlation with all other outputs.

Figure 7, Figure 8, Figure 9 and Figure 10 shows the modeling results for each dataset. As the number of sample points increases, the model accuracy gradually improves but the rate of improvement gradually decreases, as can be seen from the modeling results:

In most cases, modeling methods that consider correlation outperform independent modeling methods, and in aggregate, MBSVR works better. For example, MGP and MBSVR outperform the metric modeling approach BSVR in Output 1 for Energy_efficiency, Output 2 and Output 3 for Broomcorn, and most of the outputs for Sarcos. Except for Output 1 for Broomcorn, where the accuracy of MBSVR is significantly lower than that of the independent modeling approach (IND). In conclusion, MBSVR is more accurate in many problems than other modeling methods.

In some cases, independent modeling methods will outperform methods that consider correlations, such as polymer’s output 3 and Broomcorn’s output 1. For Broomcorn’s output 1, the independent modeling approach significantly outperforms the other. Observing the correlation coefficients, it can be found that the Pearson correlation coefficient between output 1 and output 2 is 0.39, and the correlation coefficient between output 1 and output 3 is −0.67. This disadvantage in accuracy may be due to the lack of obvious correlation, or it may be due to the model’s inability to accurately approximate the real shared information.

As the training size increases, the model accuracy advantage shows two different trends. In Output 1 and Output 4 for polymer and Output 2 and Output 3 for Broomcorn, the advantage of MBSVR over other modeling methods is more pronounced as the training size increases. However, there are also cases where the advantage of MBSVR is not obvious as the training size increases, e.g., in the Sacros dataset, the modeling accuracy of MBSVR gradually converges to the modeling accuracy of MGP as the training set increases. Theoretically, as the training set increases, the hyperparameter estimates should get closer and closer to the true values, and if the model assumptions are correct, the hyperparameter estimates are accurate, and the model accuracy improves as the training set increases.

From the results of the numerical analysis, it can be seen that in some arithmetic cases, due to the large number of hyperparameters that need to be optimized by MBSVR, the discrepancy between the shared information obtained from modeling and the actual accurate information leads to limited room for improvement in model accuracy. The final modeling accuracy is comparable to that of MGP. This discrepancy also leads to the fact that in some outputs, the accuracy of modeling methods that consider correlation may be lower than that of independent modeling methods. Additionally, MBSVR requires more optimized hyperparameters, resulting in lower time efficiency.

5. Conclusions

This paper investigates the multi-output modeling problem, aiming to improve the model accuracy by quantitatively describing the correlation between the outputs and using the information between the outputs to construct the model for multiple outputs simultaneously. To inherit the advantages of a single-output Bayesian support vector machine, based on it, the SLFM model is introduced, combined with Bayesian derivation, and the hyperparameters are optimized comprehensively to get the multi-output model that can predict multiple output means and probability distributions at the same time. Model validation is carried out on simple function arithmetic cases and real datasets, and overall, the MBSVR accuracy is higher due to the single-output modeling and the multi-output Gaussian process model.

Due to the large number of hyperparameters that need to be optimized in MBSVR, the efficiency of the algorithm is low. In addition, inaccurate hyperparameters make a difference between the shared information and the actual accurate information, resulting in limited room for improvement in model accuracy. Therefore, achieving efficient and accurate parameter optimization is the main problem that needs to be solved in the future. In addition, how to simplify the correlation description structure and further improve the applicability and optimization efficiency of the model is also the direction of further research.