Compressed Complex-Valued Least Squares Support Vector Machine Regression for Modeling of the Frequency-Domain Responses of Electromagnetic Structures

Abstract

This paper deals with the development of a Machine Learning (ML)-based regression for the construction of complex-valued surrogate models for the analysis of the frequency-domain responses of electromagnetic (EM) structures. The proposed approach relies on the combination of two-techniques: (i) the principal component analysis (PCA) and (ii) an unusual complex-valued formulation of the Least Squares Support Vector Machine (LS-SVM) regression. First, the training and test dataset is obtained from a set of parametric electromagnetic simulations. The spectra collected in the training set are compressed via the PCA by exploring the correlation among the available data. In the next step, the compressed dataset is used for the training of compact set of complex-valued surrogate models and their accuracy is evaluated on the test samples. The effectiveness and the performance of the complex-valued LS-SVM regression with three kernel functions are investigated on two application examples consisting of a serpentine delay structure with three parameters and a high-speed link with four parameters. Moreover, for the last example, the performance of the proposed approach is also compared with those provided by a real-valued multi-output feedforward Neural Network model.

1. Introduction

In recent decades, Machine Learning (ML) methods have been widely applied to construct accurate and fast-to-evaluate surrogate models able to reproduce the input–output behavior of electromagnetic (EM) structures as a function of deterministic and uncertain parameters [1,2,3,4,5,6,7,8,9,10,11,12,13,14]. In the above scenario, advanced data-driven and ML-based regressions, such as Polynomial Chaos Expansion [1,2,3,4], Support Vector Machine (SVM) regression [5,6], Least-Squares Support Vector Machine (LS-SVM) regression [7], Gaussian Process regression (GPR) [8], and feedforward [9,10,11,12], deep [12,13], convolutional [12] and Long Short-Term Memory (LSTM) [14] neural networks (NNs), have been successful applied to uncertainty quantification (UQ) and optimization in EM applications. The common idea is to adopt the above methods to train a regression model by using a limited number of training samples generated via a set of expensive full-wave or circuital simulations with the so-called computational model. Then, since the resulting surrogate model is known in a closed form, it can be suitably embedded within UQ and an optimization scheme as an efficient and effective “surrogate” of the expensive computational model [15].

Despite the high performance shown in realistic electronic and EM applications, most of the ML techniques have been developed for dealing with real-valued data [16,17,18]. However, complex-valued data are widely used in electronic applications (e.g., for AC simulations and frequency-domain analysis). The simplest strategy for extending the applicability of real-valued ML techniques to the case of complex-valued data is based on the so-called dual-channel formulation [19,20,21]. The underlaying idea is to recast the complex-valued problem into two uncorrelated real-valued ones, by stacking the real and imaginary part of the complex input and output values. The main advantage of the above procedure is that plain real-valued ML techniques can be directly adopted without requiring any generalization or improvement. However, such an approach completely ignores the possible correlation among the real and imaginary parts of the complex-valued output, thus leading to possible issues regarding accuracy and robustness to noise [19,22]. For the above reasons, alternative pure complex-valued formulations have been proposed for several ML techniques, such as NN [22], SVM regression [23], kernel Least Squares regression [19,20,21,24,25], LS-SVM regression [7] and GPR [26].

This paper investigates the performances of a pure-complex implementation of a kernel-based ML technique such as the LS-SVM regression. The proposed complex-value formulation is based on the theoretical framework developed in [19,21,24] for a plain kernel-based regression in which the regularization term is missing. Indeed, like the SVM regression, the LS-SVM regression is a powerful and well-consolidated regression technique combining the beneficial effect of the kernel and of the Tikhonov regularization [17,25,27]. Specifically, thanks to the kernel trick, the above regression allows the construction of a non-parametric model in which the number of unknowns to be estimated by the regression problem is independent from the dimensionality of the input space. Furthermore, the regularizer terms limits the overfitting issue and increases the model generalization.

The aim of this work is twofold. First, the mathematical formulation of the complex-value LS-SVM regression is developed, and the mathematical link between the pure complex formulation and the dual channel one is highlighted, where the latter is a special case of the more general complex-valued formulation. Second, the performance of compressed surrogate models constructed via the dual-channel and the pure complex-value LS-SVM regression with a complex and pseudo kernel are investigated by considering the frequency-domain responses of two test-cases consisting of a serpentine delay structure and a high-speed link, with three and four parameters, respectively. It is important to stress that, in the above applications, a compressed representation of the frequency-domain responses based on principal component analysis (PCA) [18,28,29,30] has been used to remove the redundant information and to reduce the number of LS-SVM-based models that need to be trained [31]. For the sake of completeness, for the second example, the results of the proposed approaches are compared with those provided by a real-valued multi-output feedforward NN model [11,12].

The paper is organized as follows. Section 2 describes the problem statement addressed during the paper, and its challenges. Section 3 provides an overview of the mathematical background of the PCA. Section 4 provides a self-contained formulation of the complex-valued LS-SVM regression with specific emphasis on the differences between the dual-channel and the pure complex implementation. In Section 5, the accuracy and the performance of the proposed complex-valued LS-SVM regression are investigated on two test cases consisting of a serpentine delay line structure and a transfer function of a high-speed interconnect link. Finally, the paper ends with the conclusions in Section 6.

2. Problem Statement and Challenges

Starting from a training set $D={\left\{\left(x_i,y_i\left(f_k\right)\right)\right\}}_{i,k=1}^{L,K}$ and collecting the configuration of the input parameters $x_{\mathrm{i}}={\left[x_{i,1},\dots ,x_{i,d}\right]}^T\in \mathcal{X}$, with $\mathcal{X}\in {ℂ}^d$ and the corresponding output $y_i\left(f_k\right)\in$ ℂ computed by a full-computational model (i.e., $y_i\left(f_k\right)=\mathcal{M}\left(f_k;x_i\right)$) for a set of values of the independent variable $f_k$ (e.g., the frequency), our goal is to build a surrogate model $\tilde{\mathcal{M}}$ that approximates the training data and is able to generalize well on the “unseen” data, usually known as test samples, such as:

$$y\left(f_k;x\right)\approx \tilde{\mathcal{M}}\left(f_k;x\right),$$

(1)

For any $x\in \mathcal{X}$ and for $k=1,\dots ,K$.

Without loss of generality, we seek a data-driven regression technique able to provide an accurate and efficient approximation of the actual behavior of the computational model $\mathcal{M}\left(f_k;x_i\right)$. This means that the surrogate model $\tilde{\mathcal{M}}$ should be constructed by using a small set of training samples (i.e., $L$ should be as small as possible), since the training samples are usually generated via a set of computational expensive simulations based on the full-computational model $\mathcal{M}$.

The above modeling problem is rather challenging. First, the regression technique should be able to work in the complex domain. Moreover, the resulting surrogate model should be able to mimic the behavior of a multi-output complex-valued function provided by the computational model (i.e., the output values $y_i\left(f_k\right))$, in which its values unavoidably depend on both the configuration of the input parameters $x_i$ and on the variable $f_k$.

A possible modeling scheme consists of considering the free variable $f_k$ as an extra input parameter. This means that we are seeking a “single” advanced model able to represent in a closed-form the impact of both the parameter x and the frequency $f_k$ on the system output. As an example, such a model can be obtained via a plain or recurrent NN [12,13]. However, due to the complexity of the problem at hand, the resulting neural network structure usually requires a large number of neurons and several hidden layers. Moreover, despite their flexibility, the training of NN-based structures requires the solution of a non-convex optimization leading to training issues (i.e., expensive training time and/or huge number of training samples [13,32]).

In order to partially overcome the above issues, we can think of using a single-output regression trained via the solution of a convex optimization problem, such as the plain kernel Least Squares regression [21], the LS-SVM regression [33], or the SVM regression [34], and build one model for each frequency point. In this way, the model generation requires less training samples; unfortunately, however, the overall model requires training $K$ single-output surrogates, thus making the training process extremely expensive and cumbersome when a large number of frequency points are considered.

A data compression strategy can be seen as a good compromise between the two modeling schemes depicted in the previous section. As an example, PCA [18,28,29,30] allows the extraction of the inherent correlation existing among several realizations of output data samples at different frequency points, thus leading to a compressed representation of the frequency spectra. In such a scenario, the number of actual single-output models required to represent the data can be heavily reduced.

3. PCA Compression

This section briefly presents the mathematical background of the PCA [18,28,29,30]. Let us consider the output dataset ${\left\{y_i\left(f_k\right)\right\}}_{i,k=1}^{L,K}$ that collects $L$ spectra computed for different configurations of the input parameters (i.e., the number of training outputs), each having K frequency points. The above dataset can be recast as a $K\times L$ matrix $Y$, such that the element ${\mathrm{Y}}_{i,k}=y_i\left(f_k\right)$. For normalization purposes, a zero-mean matrix is considered:

$$\tilde{Y}=\mathit{Y}-\mathit{\mu } ,$$

(2)

where $\mathit{\mu }$ is the mean value of $Y$, calculated row-wise and subtracted column-wise

The new rectangular matrix $\tilde{Y}\in {ℂ}^{K\times L}$ can be represented by means of the compact singular value decompositions (SVD) [28,30]:

$$\tilde{Y}=US{V}^{\mathrm{H}},$$

(3)

where, assuming that there are L columns of $U$ and $V$ associated with non-zero singular values, $U=\left[u_1,\dots u_L\right]\in {ℂ}^{K\times L}$ and $V=\left[{\mathit{v}}_{\mathbf{1}},\dots ,{\mathit{v}}_{\mathit{L}} \right]\in {ℂ}^{L\times L}$ are orthogonal matrices, such that $U^{\mathrm{H}}U=V^{\mathrm{H}}V={\mathit{I}}_{L\times L}$, collecting the left and right singular vectors, and $\mathrm{S}=\mathrm{diag}\left\{\left({\mathsf{\sigma }}_1,\dots {\mathsf{\sigma }}_{\mathrm{L}}\right)\right\}\in {ℂ}^{L\times L}$ is a diagonal matrix collecting the singular values sorted in a decreasing order ${\sigma }_1\gg {\sigma }_2\gg \cdots \gg {\sigma }_L$. Now, a compressed approximation of the actual matrix $\tilde{Y}$ can be obtained as follows:

$$\tilde{Y}\approx \tilde{U}\tilde{S}{\tilde{V}}^{\mathrm{H}},$$

(4)

where $\tilde{U}=\left[u_1,\dots ,u_n\right]\in {ℂ}^{K\times \tilde{\mathrm{n}}}$ with $u_i\in {ℂ}^{K\times 1}$ and $\tilde{V}=\left[v_1,\dots ,v_n\right]\in {ℂ}^{\tilde{n}\times L}$ with $v_i\in {ℂ}^{L\times 1}$ are the reduced left and right-eigenvector matrices collecting only the first $\tilde{n}$ components (i.e., the first $\tilde{n}$ columns of the original matrices $U$ and $V$), and $\tilde{S}=diag\left\{\left({\sigma }_1,\dots ,{\sigma }_{\tilde{n}}\right)\right\}$ is a reduced diagonal matrix containing the first $\tilde{n}$ singular values.

The above relationship can be used to obtain a compressed representation $Z\in {ℂ}^{\tilde{n}\times L }$ of the original matrix $Y$, such that:

$$Z={\tilde{U}}^H\tilde{Y}=\tilde{S}{\tilde{V}}^{\mathrm{H}},$$

(5)

It is important to note the resulting compressed matrix $Z\in {ℂ}^{\tilde{n}\times L}$ is smaller than the original matrix $\tilde{Y}$, since usually $\tilde{n}\ll K$. Moreover, the rows of the compressed matrix $Z$ can be considered to be the realizations of a new set of output variables ${\left\{z\left(x_l\right)\right\}}_{l=1}^L$, with $\mathit{z}\left(x_l\right)\in {ℂ}^{\tilde{n}\times 1}$, which can be considered as the collection of $L$ samples of a compressed $\tilde{n}$-dimensional output variable. Specifically, the element ($i,j$) of the matrix $Z$, corresponds to the $i$-th output of the compressed representation evaluated at the j-th configuration of the input parameters, i.e., $Z_{ij}=z_i\left(x_j\right)$. This means that only $\tilde{n}$ single-output models need to be trained to represent the whole dataset provided by the matrix Y, thus leading to a substantial improvement in the training time. Once a surrogate model for each of the $\tilde{n}$ components of the compressed multi-output representation in $Z$ is available, the overall compress surrogate can be inexpensively used to predict the system output $y\left(\overline{x}\right)\in {ℂ}^{K\times 1}$ for a generic test sample $\overline{x}\in \mathcal{X}$ as follows:

$$y\left(\overline{x}\right)\approx \mathbf{\mu }+\tilde{U}\overline{\mathit{Z}} ,$$

(6)

where $\overline{\mathit{Z}}={\left[z_1\left(\overline{x}\right),\dots ,z_{\tilde{n}}\left(\overline{x}\right)\right]}^{\mathrm{T}}\in {ℂ}^{\tilde{n}\times 1}$.

It is important to remark that the compressed representation $\mathit{z}\left(x\right)$ provided by the PCA compression allows approximating the actual data $y\left(x\right)$ with a tunable accuracy depending on the number of PCA components $\tilde{n}$, such that [28]:

$${\left(\frac{{\sigma }_{\tilde{n}+1}}{{\sigma }_1}\right)}^2\le {\epsilon }^2,$$

(7)

where $\epsilon$ is a given error threshold tuned by the user.

4. Complex Valued Least-Square Support Vector Machine Regression

The LS-SVM regression is a kernel-based regression with an L2 regularizer, which allows constructing a non-parametric model in which the number of unknowns and complexity is independent from the number of input parameters via the solution of a simple linear system, i.e., by solving a convex optimization problem [33].

However, the plain formulation of such a technique has been developed for real-valued datasets only. The aim of this section is to extend the mathematical formulation of the LS-SVM regression to the more general case of the complex-data problem. The proposed formulation is based on the preliminary results presented in [25] and on the results presented in [19,21] for a classical reproducing kernel Hilbert space (RKHS) regression, in which the regularizer term is neglected. For the sake of simplicity, the following complex valued formulation is developed for the simplified case of single-output regression, but all the calculations can be easily extended to the case of a multi-output problem.

Starting from a set of complex-valued samples $\mathcal{D}={\left\{\left(x_l,y_l\right)\right\}}_{l=1}^L$ where $x_l\in {ℂ}^d$ and $y_l=y(x_l)\in ℂ$, the primal space formulation of the LS-SVM regression is written:

$$y\approx \tilde{\mathcal{M}}\left(x\right)={\displaystyle \sum }_{i=1}^N{w}_i{\varphi }_i^{\ast }\left(x\right)+b=w,\Phi \left(x\right)+b,$$

(8)

where $w={\left[w_1,\dots ,w_N\right]}^T=w_R+j{w}_I\in {ℂ}^N$ are the complex regression coefficients (i.e., $w_i=w_{i,R}+j{w}_{i,I}$), $\Phi \left(x\right)={\Phi }_R\left(x\right)+j{\Phi }_{\mathrm{I}}\left(x\right)={\left[{\varphi }_1\left(x\right),\dots ,{\varphi }_N\left(x\right)\right]}^T$ is a vector-valued complex function $\Phi \left(\cdot \right):{ℂ}^d\to {ℂ}^N$ collecting the complex-valued basis functions ${\varphi }_{\mathrm{i}}\left(x\right)$ that provide a map between the parameter space and the feature space, $b=b_R+j{b}_I$ is the bias term, and $w,\Phi \left(x\right)={\Phi }^{\mathrm{H}}\left(x\right)w=w^T{\Phi }^{\ast }\left(x\right)$ is the inner product in the complex domain.

The primal space formulation of the complex-valued regression in Equation (8) can be written in term of its real and imaginary components:

$$\tilde{\mathcal{M}}\left(x\right)={\tilde{\mathcal{M}}}_R\left(x\right)+j{\tilde{\mathcal{M}}}_I\left(x\right)=\left({\mathit{w}}_R{\Phi }_R^T\left(x\right)+w_I{\Phi }_I^T\left(x\right)+b_R\right)+j\left(w_{\mathrm{I}}{\Phi }_R^T\left(x\right)-{\mathit{w}}_{R }{\Phi }_I^T\left(x\right)+b_i\right) ,$$

(9)

In the above primal space formulation, the regression unknowns (i.e., the coefficients in $w_R$ and $w_I$, and the bias term $b_R$ and $b_I$, respectively) are estimated by solving the following convex optimization problem:

$$\underset{w_R,w_I,b_R,b_I,e_R,e_I}{\min }\frac{1}{2}{w}_R^T{w}_R+\frac{1}{2}{w}_I^T{w}_I+\frac{{\gamma }_R}{2}{\displaystyle \sum }_{l=1}^L{e}_{R,l}^2+\frac{{\gamma }_I}{2}{\displaystyle \sum }_{l=1}^L{e}_{I,l}^2$$

(10)

Such that

$$e_{\mathit{R},\mathit{l}}=\mathit{R}\mathit{e}\left\{{\mathit{y}}_l-\tilde{\mathcal{M}}\left(x_l\right)\right\}={\mathit{y}}_{\mathit{R},\mathit{l} }-\left(w_{\mathit{R}}{\Phi }_{\mathit{R}}\left(x_{\mathit{l}}\right)+w_{\mathit{I}}{\Phi }_{\mathit{I}}\left(x_{\mathit{l}}\right)+{\mathit{b}}_{\mathit{R}}\right)$$

$${\mathit{e}}_{\mathit{R},\mathit{l}}=\mathit{I}\mathit{m}\left\{{\mathit{y}}_{\mathit{l}}-\tilde{\mathcal{M}}\left(x_l\right)\right\}={\mathit{y}}_{\mathit{I},\mathit{l} }-(w_{\mathit{I}}{\Phi }_{\mathit{R}}\left(x_{\mathit{l}}\right)-w_{\mathit{R}}{\Phi }_{\mathit{I}}\left(x_{\mathit{l}}\right)+{\mathit{b}}_{\mathit{I}})$$

For $l=1,\dots , L,$ where ${\mathit{w}}_{\mathit{R}}^{\mathit{T}}{\mathit{w}}_{\mathit{R}}+{\mathit{w}}_{\mathit{I}}^{\mathit{T}}{\mathit{w}}_{\mathit{I}}={\mathit{w}}_{\mathit{2}}^{\mathit{2}}$ is the L2 regularizer and the terms $e_{R,l}^2$ and $e_{I,l}^2$ provide the error of a squared loss function.

The Lagrangian for the above constraint optimization problem is written:

$$\begin{array}{c}\mathcal{L}({\mathit{w}}_R,{\mathit{w}}_I,b_R,b_I,{\mathit{e}}_R,{\mathit{e}}_I,{\mathit{\alpha }}_R,{\mathit{\alpha }}_I)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=\frac{1}{2}{\mathit{w}}_R^T{\mathit{w}}_R+\frac{1}{2}{\mathit{w}}_I^T{\mathit{w}}_I+\frac{{\gamma }_R}{2}{\displaystyle \sum _{l=1}^L}{e}_{R,l}^2+\frac{{\gamma }_I}{2}{\displaystyle \sum _{l=1}^L}{e}_{I,l}^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-{\displaystyle \sum _{l=1}^L}{\alpha }_{R,l}\left\{e_{R,l}-y_{R,l }+\left(w_R^T{\Phi }_R\left({\mathrm{x}}_l\right)+w_I^T{\Phi }_I\left({\mathrm{x}}_l\right)+b_R\right)\right\}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-{\displaystyle \sum _{l=1}^L}{\alpha }_{I,l}\left\{e_{I,l}-y_{I,l }+\left(w_I^T{\Phi }_R\left({\mathrm{x}}_l\right)-w_R^T{\Phi }_I\left({\mathrm{x}}_l\right)+b_I\right)\right\} ,\end{array}$$

(11)

where ${\alpha }_l={\alpha }_{R,l}+j{\alpha }_{I,l}$ are the Lagrangian multipliers such that ${\alpha }_{R,l},{\alpha }_{I,l}\ge 0$ for $l=1,\dots , L.$

By computing the partial derivatives of the Lagrangian $\mathcal{L}\left({\mathit{w}}_R,{\mathit{w}}_I,b_R,b_I,{\mathit{e}}_R,{\mathit{e}}_I,{\mathit{\alpha }}_R,{\mathit{\alpha }}_I\right)$ with respect to its parameters:

$$\frac{\partial \mathcal{L}}{\partial w_R}=0\to w_R={\displaystyle \sum }_{l=1}^L\left[{\alpha }_{R,l}{\Phi }_R\left({\mathrm{x}}_l\right)-{\alpha }_{I,l}{\Phi }_I\left({\mathrm{x}}_l\right)\right],$$

(12)

$$\frac{\partial \mathcal{L}}{\partial w_I}=0\to w_I={\displaystyle \sum }_{l=1}^L\left[{\alpha }_{R,l}{\Phi }_I\left({\mathrm{x}}_l\right)+{\alpha }_{I,l}{\Phi }_R\left({\mathrm{x}}_l\right)\right],$$

(13)

$$\frac{\partial \mathcal{L}}{\partial b_R}=0\to {\displaystyle \sum }_{l=1}^L{\alpha }_{R,l}=0 ,$$

(14)

$$\frac{\partial \mathcal{L}}{\partial b_I}=0\to {\displaystyle \sum }_{l=1}^L{\alpha }_{I,l}=0 ,$$

(15)

$$\frac{\partial \mathcal{L}}{\partial e_{R,l}}=0\to {\gamma }_R{e}_{R,l}={\alpha }_{R,l} ,$$

(16)

$$\frac{\partial \mathcal{L}}{\partial e_{I,l}}=0\to {\gamma }_I{e}_{I,l}={\alpha }_{I,l} ,$$

(17)

$$\frac{\partial \mathcal{L}}{\partial {\alpha }_{R,l}}=0\to e_{R,l}-y_{R,l }+\left(w_R^T{\Phi }_R\left({\mathrm{x}}_l\right)+w_I^T{\Phi }_I\left({\mathrm{x}}_l\right)+b_R\right)=0,$$

(18)

$$\frac{\partial \mathcal{L}}{\partial {\alpha }_{I,l}}=0\to e_{I,l}-y_{I,l }+\left(w_I^T{\Phi }_R\left({\mathrm{x}}_l\right)-w_R^T{\Phi }_I\left({\mathrm{x}}_l\right)+b_I\right)=0,$$

(19)

For $l=1,\dots ,L.$

By substituting (12)–(17) in (18) and (19):

$$\left\{\begin{array}{c}\frac{{\alpha }_{R,l}}{{\gamma }_R}-y_{R,l }+{\displaystyle \sum }_{i=1}^L{\left[{\alpha }_{R,i}{\Phi }_{\mathrm{R}}\left(x_i\right)-{\alpha }_{I,i}{\Phi }_I\left(x_i\right)\right]}^T{\Phi }_R\left(x_l\right)+{\displaystyle \sum }_{i=1}^L{\left[{\alpha }_{R,i}{\Phi }_{\mathrm{I}}\left(x_i\right)+{\alpha }_{I,i}{\Phi }_R\left(x_i\right)\right]}^T{\Phi }_I\left(x_l\right)+b_R=0\\ \frac{{\alpha }_{I,l}}{{\gamma }_I}-y_{I,l }+{\displaystyle \sum }_{i=1}^L{\left[{\alpha }_{R,i}{\Phi }_{\mathrm{I}}\left(x_i\right)+{\alpha }_{I,i}{\Phi }_R\left(x_i\right)\right]}^T{\Phi }_R\left(x_l\right)-{\displaystyle \sum }_{i=1}^L{\left[{\alpha }_{R,i}{\Phi }_{\mathrm{R}}\left(x_i\right)-{\alpha }_{I,i}{\Phi }_I\left(x_i\right)\right]}^T{\Phi }_I\left(x_l\right)+b_I=0\\ {\displaystyle \sum }_{l=1}^L{\alpha }_{R,l}=0\\ {\displaystyle \sum }_{l=1}^L{\alpha }_{I,l}=0\end{array}\right.$$

(20)

For $l=1,\dots ,L.$

The above system of equations is the dual-form representation of the optimization problem in Equation (10), in which the original regression coefficients collected in the vectors $w_R$ and $w_I$ are replaced by the Lagrangian multipliers ${\alpha }_R$ and ${\alpha }_I$. It is important to remark that, although the number of unknowns in the primal space (i.e., the dimensionality of $\left|w\right|=N$) is given by the number of basis functions, in the dual formulation the number of unknowns (i.e., the Lagrangian multipliers collected in the vectors $\mathbf{\alpha }={\alpha }_R+j{\alpha }_I$) is always equal to the number of the training samples $L$. This means the resulting model is non-parametric, i.e., a model in which its complexity is independent from the number of both the input parameters and the basis functions.

In order to define the kernel function and the dual formulation of the complex-valued LS-SVM, the first two equations in Equation (20) can be rewritten as follows:

$$\frac{{\alpha }_{R,l}}{{\gamma }_R}-y_{R,l }+{\displaystyle \sum }_{i=1}^L{\alpha }_{R,i}\left[{\Phi }_{\mathrm{R}}^T\left(x_i\right){\Phi }_R\left(x_l\right)+{\Phi }_{\mathrm{I}}^T\left(x_i\right){\Phi }_{\mathrm{I}}\left(x_l\right)\right]+{\displaystyle \sum }_{i=1}^L{\alpha }_{I,i}\left[{\Phi }_R^{\mathrm{T}}\left(x_i\right){\Phi }_I\left(x_l\right)-{\Phi }_{\mathrm{I}}^T\left(x_i\right){\Phi }_R\left(x_l\right)\right]+b_R=0,$$

(21)

$$\frac{{\alpha }_{I,l}}{{\gamma }_I}-y_{I,l }+{\displaystyle \sum }_{i=1}^L{\alpha }_{R,i}\left[{\Phi }_{\mathrm{I}}^{\mathrm{T}}\left(x_i\right){\Phi }_R\left(x_l\right)-{\Phi }_{\mathrm{R}}^{\mathrm{T}}\left(x_i\right){\Phi }_I\left(x_l\right)\right]+{\displaystyle \sum }_{i=1}^L{\alpha }_{I,i}\left[{\Phi }_R^{\mathrm{T}}\left(x_i\right){\Phi }_R\left(x_l\right)+{\Phi }_I^T\left(x_i\right){\Phi }_I\left(x_l\right)\right]+b_I=0,$$

(22)

For $l=1,\dots ,L.$

Now, let us define a complex-valued kernel $k_c\left(x,x^{\prime }\right)$:

$$\begin{array}{c}{\mathit{k}}_{\mathit{c}}\left(x,x^{\prime }\right)=\Phi \left(x\right),\Phi \left(x^{\prime }\right)={\Phi }^T\left(x\right)\cdot {\Phi }^{\ast }\left(x^{\prime }\right)={\left({\Phi }_{\mathit{R}}\left(x\right)+\mathit{j}{\Phi }_{\mathit{I}}\left(x\right)\right)}^{\mathit{T}}\cdot \left({\Phi }_{\mathit{R}}\left(x^{\prime }\right)-\mathit{j}{\Phi }_{\mathit{I}}\left(x^{\prime }\right)\right)=\\ \left[{\Phi }_{\mathit{R}}^{\mathit{T}}\left(x\right){\Phi }_{\mathit{R}}\left(x^{\prime }\right)+{\Phi }_{\mathit{I}}^{\mathit{T}}\left(x\right){\Phi }_{\mathit{I}}\left(x^{\prime }\right)\right]+\mathit{j}\left[{\Phi }_{\mathit{I}}^{\mathit{T}}\left(x\right){\Phi }_R\left(x^{\prime }\right)-{\Phi }_{\mathit{R}}^{\mathit{T}}\left(x\right){\Phi }_{\mathit{I}}\left(x^{\prime }\right)\right]={\mathit{k}}_{ℝ}\left(x,x^{\prime }\right)+\mathit{j}{\mathit{k}}_{\mathbb{I}}\left(x,x^{\prime }\right)\end{array}$$

(23)

Similar to the real-valued formulation, the kernel function is defined as the inner product in the complex feature space of the basis functions evaluated at $x$ and $x’$, where: $k_{ℝ}\left(x,x^{\prime }\right)={\Phi }_R^T\left(x\right){\Phi }_R\left(x^{\prime }\right)+{\Phi }_{\mathrm{I}}^T\left(x\right){\Phi }_I\left(x^{\prime }\right)$ and $k_{\mathbb{I}}\left(x,x^{\prime }\right)={\Phi }_I^T\left(x\right){\Phi }_{\mathrm{R}}\left(x^{\prime }\right)-{\Phi }_{\mathrm{R}}^T\left(x\right){\Phi }_I\left(x^{\prime }\right).$

Using the above definition, Equations (21) and (22) can be rewritten as follows:

$$\frac{{\alpha }_{R,l}}{{\gamma }_R}-y_{R,l}+{\displaystyle \sum }_{i=1}^L\left[{\alpha }_{R,i}{k}_{ℝ}\left(x_i,x_l\right)-{\alpha }_{I,i}{k}_{\mathbb{I}}\left(x_i,x_l\right)\right]+b_R=0$$

(24)

$$\frac{{\alpha }_{I,l}}{{\gamma }_I}-y_{I,l}+{\displaystyle \sum }_{i=1}^L\left[{\alpha }_{I,i}{k}_{ℝ}\left(x_i,x_l\right)+{\alpha }_{R,i}{k}_{\mathbb{I}}\left(x_i,x_l\right)\right]+b_I=0$$

(25)

For $l=1,\dots ,L.$

In the above equations, the regression unknowns can be computed by solving the following linear system:

$$\left[\begin{array}{cccc}{\mathit{K}}_{RR}+\frac{{\mathit{I}}_L}{{\gamma }_R}& {\mathit{K}}_{RI}& \mathbf{1}& \mathbf{0}\\ {\mathit{K}}_{IR}& {\mathit{K}}_{II}+\frac{{\mathit{I}}_L}{{\gamma }_I}& \mathbf{0}& \mathbf{1}\\ {\mathbf{1}}^T& {\mathbf{0}}^T& 0& 0\\ {\mathbf{0}}^T& {\mathbf{1}}^T& 0& 0\end{array}\right]\left[\begin{array}{c}{\mathit{\alpha }}_R\\ {\mathit{\alpha }}_I\\ b_R\\ b_I\end{array}\right]=\left[\begin{array}{c}{y}_R\\ y_I\\ 0\\ 0\end{array}\right]$$

(26)

where ${\mathit{I}}_L$ is the identity matrix of size $L\times L$, ${\mathbf{1}}^T={\left[1,\dots ,1\right]}^T\in {ℝ}^{1\times L}$, ${\mathbf{0}}^{\mathit{T}}={\left[0,\dots ,0\right]}^T\in {ℝ}^{1\times L}$, ${\mathit{\alpha }}_R, {\mathit{\alpha }}_I\in {ℝ}^{L\times 1}$ are vectors collecting the real and imaginary part of the regression coefficients, $b=b_R+j{b}_I$ is the bias term and ${\mathit{K}}_{RR}, {\mathit{K}}_{RI},{\mathit{K}}_{IR}, {\mathit{K}}_{II}\in {ℝ}^{L\times L}$ are kernel matrices defined as:

$$K_{RR}^{\left(i,j\right)}=K_{II}^{\left(i,j\right)}=k_{ℝ}\left(x_i,x_j\right)$$

(27)

$$K_{IR}^{\left(i,j\right)}=-K_{RI}^{\left(i,j\right)}=k_{\mathbb{I}}\left(x_i,x_j\right)$$

(28)

for any $i,j=1,\dots ,L$. The parameters ${\gamma }_R$ and ${\gamma }_I$ are the regularizer hyperparameters tuned by the user and provide a trade-off between the model flatness and its accuracy [34].

It is important to remark that the above linear system is square and, therefore, if the determinant of the square matrix is different from 0, it always yields a unique solution, which leads to the following dual space formulation for complex-valued of LS-SVM regression:

$$\mathrm{y}\left(\mathrm{x}\right)={\displaystyle \sum }_{l=1}^L{\alpha }_l{\mathrm{k}}_C\left(x_l,x\right)+b$$

(29)

By substituting Equation (23) in the above equation gives:

$$\mathrm{y}\left(x\right)={\displaystyle \sum }_{l=1}^L\left[\left({\alpha }_{R,l}{\mathrm{k}}_{ℝ}\left(x_l,x\right)-{\alpha }_{I,l}{\mathrm{k}}_{\mathbb{I}}\left(x_l,x\right)+b_R\right)+j\left({\alpha }_{R,l}{\mathrm{k}}_{\mathbb{I}}\left(x_l,x\right)+{\alpha }_{I,l}{\mathrm{k}}_{ℝ}\left(x_l,x\right)+b_I\right)\right]$$

(30)

From the above formulation, it is clear that by means of the complex kernel ${\mathrm{k}}_C$, the complex-valued LS-SVM regression in the dual space is able to account for possible correlation between the real and imaginary part of $\mathrm{y}\left(x\right)$.

4.1. Complex-Valued Kernel

There are several strategies to construct a complex kernel $k_C$. Within this paper, we will investigate two of them, the independent kernel, referred to as the complex valued complex function (CVCF) [19], and the pseudo kernel, referred to as the pseudo complex-valued function (PCF) [35,36]. A generic CVCF kernel can be constructed starting from a real-valued kernel $k_{ℝ}$ as follows:

$${\mathrm{k}}_{\mathrm{C}}\left(x,x^{\prime }\right)={\mathrm{k}}_{ℝ}\left(x_R,x_{\mathrm{R}}^{\prime }\right)+{\mathrm{k}}_{ℝ}\left(x_I,x_I^{\prime }\right)+j\left({\mathrm{k}}_{ℝ}\left(x_R,x_I^{\prime }\right)+{\mathrm{k}}_{ℝ}\left(x_I,x_{\mathrm{R}}^{\prime }\right)\right),$$

(31)

The above complex kernel is fully compliant with the definition provided in Equation (23). The real kernel ${\mathrm{k}}_{ℝ}$ can be any real kernel function, e.g., linear kernel, radial basis function (RBF) kernel and polynomial kernel. Hereafter, in this paper, for the CVCF kernel we will use ${\mathrm{k}}_{ℝ}$ as the RBF kernel, i.e.,

$${\mathrm{k}}_{ℝ}\left(x,x^{\prime }\right)=\exp \left(-\frac{1}{2{\sigma }^2}x-x\prime {)}^2\right),$$

(32)

where $\sigma$ is the kernel hyperparameter, which in this work will be tuned, along with the regularizer hyperparameters, during the model training by combining cross validation (CV) with a Bayesian optimizer [37,38].

As an alternative, a family of kernels based on the PCF can be suitably generated from the isotropic complex covariance function, such that (additional mathematical details are provided in [19,36]):

$${\mathrm{k}}_{\mathrm{c}}\left(x,x\prime \right)=\cos \left(cx-x^{\prime }\right)k_{ℝ}\left(x,x^{\prime }\right)+\mathrm{j} \sin \left(cx-x\prime \right){\mathrm{k}}_{ℝ}\left(x,x^{\prime }\right),$$

(33)

where ${\mathrm{k}}_{ℝ}\left(x,x_{\mathit{k}}\right)$ can be selected as any kernel function and $c$ is a new hyperparameter. In this specific case, a Rational Quadratic kernel is adopted, such as ${\mathrm{k}}_{ℝ}\left(x,x\prime \right)=$ ${\sigma }^2\left(1+\frac{x-x{\prime }^2}{2a{l}^2}\right){}^{-a}$, which is a Rational Quadratic kernel in which a, l and $\sigma$ are additional hyperparameters. Moreover, in this case all the hyperparameters are tuned via CV and a Bayesian optimizer [37,38].

4.2. Dual Channel Kernel (DCK) LS-SVM for Complex-Valued Data

The dual channel kernel (DCK) formulation can be seen as a special case of the general mathematical framework presented in the previous section. The underlying idea is to recast the complex variables in terms of their real and imaginary part and to work with a standard real kernel, i.e., $k_c=k_{ℝ}$: ${ℝ}^{d\times d}\to ℝ$.

Under the above assumption, the regression problem in Equation (26) can be simplified as follows:

$$\left[\begin{array}{cccc}{\mathit{K}}_{RR}+\frac{{\mathit{I}}_L}{{\gamma }_R}& {\mathbf{0}}_L& \mathbf{1}& \mathbf{0}\\ {\mathbf{0}}_{\mathit{L}}& {\mathit{K}}_{RR}+\frac{{\mathit{I}}_L}{{\gamma }_I}& \mathbf{0}& \mathbf{1}\\ {\mathbf{1}}^{\mathit{T}}& {\mathbf{0}}^{\mathit{T}}& 0& 0\\ {\mathbf{0}}^{\mathit{T}}& {\mathbf{1}}^{\mathit{T}}& 0& 0\end{array}\right]\left[\begin{array}{c}{\mathit{\alpha }}_{\mathit{R}}\\ {\mathit{\alpha }}_{\mathit{I}}\\ b_R\\ b_I\end{array}\right]=\left[\begin{array}{c}{y}_{\mathit{R}}\\ y_{\mathit{I}}\\ 0\\ 0\end{array}\right]$$

(34)

where ${\mathbf{0}}_{\mathit{L}}$ is the $L\times L$ null matrix, ${\mathit{K}}_{RR}\in {ℝ}^{L\times L}$ such that $K_{RR}^{\left(i,j\right)}=k_{ℝ}\left(x_i,x_j\right)$, whilst the matrices ${\mathit{K}}_{IR}=-{\mathit{K}}_{RI}={\mathbf{0}}_L$.

It is important to remark that, in the above formulation, there is no coupling between the real and imaginary coefficients ${\mathit{\alpha }}_R$ and ${\mathit{\alpha }}_I$, and bias terms $b_R$ and $b_I$. Indeed, the solution of the above linear system is equivalent to solving two decoupled ones, accounting for the real and imaginary parts of the regression unknowns independently, such as:

$$\left[\begin{array}{cc}{\mathit{K}}_{RR}+\frac{{\mathit{I}}_{\mathit{L}}}{{\gamma }_R}& \mathbf{1}\\ {\mathbf{1}}^T& 0\end{array}\right]\left[\begin{array}{c}{\mathit{\alpha }}_{\mathit{R}}\\ b_R\end{array}\right]=\left[\begin{array}{c}{\mathit{y}}_{\mathit{R}}\\ 0\end{array}\right],$$

(35)

$$\left[\begin{array}{cc}{\mathit{K}}_{RR}+\frac{{\mathit{I}}_{\mathit{L}}}{{\gamma }_I}& \mathbf{1}\\ {\mathbf{1}}^T& 0\end{array}\right]\left[\begin{array}{c}{\mathit{\alpha }}_{\mathit{I}}\\ b_I\end{array}\right]=\left[\begin{array}{c}{\mathit{y}}_{\mathit{I}}\\ 0\end{array}\right],$$

(36)

In this work, a standard RBF kernel is considered as the real kernel $k_{ℝ}\left(x_i,x_j\right)$. In the above scenario, the dual space formulation of the LS-SVM is written [27]:

$${\mathrm{y}}_{\mathrm{R}}\left(x\right)={\displaystyle \sum }_{l=1}^L{\alpha }_{R,l}{k}_{ℝ}\left(x_l,x\right)+b_R,$$

(37)

$${\mathrm{y}}_{\mathrm{I}}\left(x\right)={\displaystyle \sum }_{l=1}^L{\alpha }_{I,l}{k}_{ℝ}\left(x_l,x\right)+b_I,$$

(38)

It is important to note that, in the above formulation, the model for the real and imaginary parts of $y$ are built separately, thus ignoring any possible correlation between them. Furthermore, the above models can be trained using the LS-SVM Lab toolbox for the LS-SVM available in MATLAB [39].

5. Application Examples

This section compares the accuracy and the robustness against noise of the three implementations of the complex-valued LS-SVM regression provided in Section 4 by considering two different application examples. Specifically, the proposed approaches are applied to predict the scattering parameters of a serpentine structure with three parameters and the transfer function of a high-speed link with four parameters.

5.1. Example I

As a first test case, the complex-valued LS-SVM regression applied to calculate the scattering parameters of a serpentine delay line structure is presented. Serpentine lines are widely used in printed circuit board (PCB) design to compensate time delays introduced by the trace routing. However, the frequency-domain behavior of such a structure is heavily affected by its geometrical and electrical parameters and should be carefully assessed during the design phase to avoid signal and power integrity issues and to meet design constraints [40].

The structure of the serpentine delay line considered in this example is shown in Figure 1 (inspired by [40]). The S21 scattering parameters of the above structure are investigated as a function of three parameters (i.e., $x={\left[{\epsilon }_r,LL,SW\right]}^T$) in a frequency bandwidth from 1 MHz to 1 GHz.

Table 1. Serpentine delay line parameters for the training and test dataset.

Training Ranges	Test Ranges
4.5 mm ≤ LL ≤ 5.1 mm	4.5 mm ≤ LL ≤ 5.1 mm
3.9 ≤ εr ≤ 4.5	3.9 ≤ εr ≤ 4.5
0.13 mm ≤ SW ≤ 0.17 mm	0.13 mm ≤ SW ≤ 0.17 mm
1 MHz ≤ f ≤ 3 GHz	1 MHz ≤ f ≤ 3 GHz
1000 samples for each frequency	2000 samples for each frequency

shows the range of variability serpentine delay line parameters that are simulated to produce the training and test data.

The whole dataset consists of 3000 samples (1000 training data and 2000 test samples). The samples were generated via Latin Hypercube Sampling (LHS) by assuming a uniform variability between their maximum and minimum value. For each configuration of the geometrical parameters, the corresponding scattering parameters were computed for 5000 linearly spaced frequency sample points. The samples were generated via a set of parametric simulations with the full-wave solver available in CST.

The PCA compression is then used to compress the data in the frequency domain and to reduce the computation for the model training. Figure 2 shows the behavior normalized singular values of the frequency points of the training dataset. The plot shows that $\overline{n}$ = 9 is enough to represent the whole training set with a 0.001% threshold.

After applying the PCA, the performances of the LS-SVM regression using the PCF and the CVCF complex kernel function (see Section 4.1), and of the DCK LS-SVM regression (see Section 4.2), were assessed on the test samples for the S21 parameter. To investigate the performance of the mentioned methods, the relative Root Mean Square Error (NRMSE) is calculated as the following equation:

$$NRMSE \%=100\cdot \frac{\frac{1}{T}\sqrt{{{\displaystyle \sum }}_{t=1}^T{\left(X_y-X_{\widehat{y}}\right)}^2} }{\frac{1}{T}\sqrt{{{\displaystyle \sum }}_{t=1}^T{X}_y^2}},$$

(39)

where $X_y$ can be either the real or imaginary part of the actual test samples, $X_{\widehat{y}}$ is the corresponding prediction estimated via the proposed metamodels, and $T$ is the number of test samples.

Figure 3 and Figure 4 show the normalized error for real and imaginary parts of these three regression approaches for an increasing number of training samples (i.e., L = 50, 250, and 500).

The plots highlight the improved accuracy of the CVCF and PCF with respect to the DCK. Indeed, with the DCK-based model, we are implicitly neglecting any kind of correlation between the real and imaginary parts of the S parameters, and this lack of complexity becomes even more evident when a low number of training samples is used to train the model.

5.2. Example II

The second application example is based on the high-speed link, as depicted in Figure 5, representing a signal distribution on a PCB. Similar to the previous example, the frequency response of the link, and thus its performance, can be greatly influenced by possible variations of its internal parameters [5].

Specifically, the proposed modeling approaches are here adopted to build a surrogate model for the frequency-domain behavior of the complex-valued transfer function, in which $y\left(x;f\right)=\frac{\left.Vout(f;x\right)}{\left.E(f\right)}$, as a function of the values of four lumped components $C_1\left(x_1\right)$, $C_2\left(x_2\right)$, $L_1\left(x_3\right)$, and $L_2\left(x_4\right)$, is defined by four uniformly distributed random variables with a variation of ±50% around their central value collected in $x={\left[x_1,x_2,x_3,x_4\right]}^T$ (additional details are provided in

Table 2. High-speed interconnect link parameters for the training and test datasets.

Training and Test Ranges
C1 (x1)	(1 ± 0.5 x1) pF
C2 (x2)	(0.5 ± 0.25 x2) pF
L1 (x3)	(10 ± 5 x3) nH
L2 (x4)	(10 ± 5 x4) nH
i	1,2,3,4
xi	Random variable in [−1, 1]

).

Four sets consisting of L = 20, 100, 150, and 500 training input configurations were generated via an LHS and used as input for a computational model implemented in MATLAB, which provides as output the corresponding transfer function evaluated at 200 frequency points. The PCA is applied to remove redundant information, leading to a compressed representation of the original dataset with only $\overline{n}$ = 10 components using a threshold of 0.01%.

The compressed training sets are then used to train three different surrogate models based on the DCK, PCF, and CVCF LS-SVM regressions. For the sake of completeness, the predictions of the above methods are compared with those provided by an additional surrogate model built via a multi-output feedforward NN structure [34,35] considering the real and imaginary parts of the considered transfer function (i.e., using the dual channel implementation). The NN is trained via the Gradient Decent backpropagation method implemented within the Neural Net Fitting Tool available in the MATLAB Deep Learning Toolbox. The network consists of three hidden layers with 50, 20, and 15 neurons, respectively. The activation function is the hyperbolic tangent sigmoid.

Figure 6 and Figure 7 show the performance of each method on a test set consisting of 1000 samples assessed via the relative NRMSE in Equation (39) for the real and imaginary parts, respectively. The results clearly highlight the improved accuracy achieved via the PCF. Moreover, as expected, due to its simplified formulation, the DCK again provides the lowest accuracy. By comparison, NN shows the lower accuracy when a small number of training samples is used (i.e., up to 150 training samples). This low convergence with respect to the number of training samples is due to the inherent non-convex nature of the optimization problem solved during the model training [32].

Moreover, in order to stress the reliability of the proposed techniques, the training output $\mathrm{y}$ was corrupted with Gaussian noise, such that:

$$y_{i,noisy}\left(x_{\mathrm{i}}\right)=y_i\left(x_{\mathrm{i}}\right)\times \left(1+{\zeta }_n\right),$$

(40)

where ${\zeta }_n\sim \mathcal{N}\left(0,{\sigma }_n^2\right)$ is a Gaussian random variable with standard deviation ${\sigma }_n=\left[0.01, 0.03\right]$.

Figure 8 and Figure 9 compare the relative NRMSE computed at a single frequency point selected as the one providing the maximum error via the proposed approaches, and a multi-output feedforward NN for different values of the noise standard deviation ${\sigma }_n$ and the number of training samples L for the real and imaginary parts, respectively. Among the introduced methods, CVCF shows the better performance and robustness against noise, both for real and imaginary parts.

6. Conclusions

In this paper, the application of different methods based on LS-SVM regression for predicting the complex-valued frequency response was presented in a serpentine delay line and a high-speed link. The required data for training and examination were obtained from CST and MATLAB software, in terms of physical and geometrical parameters, including substrate dielectric constant and height and strip width of the serpentine delay line, and the capacitances and inductances of the high-speed link. PCA is shown to reduce the computational cost of the frequency samples. Then, the training data are used to create the model, and the test data are considered for the examination of the established model. The performance of the three proposed methods and of a multi-output feedforward NN are compared on the test dataset. It is shown that the CVCF method is suitable for the first complex electromagnetic problem when there are few training data points. In addition, PCF shows an acceptable performance under noise-free conditions at all frequency points for the second example. Finally, the results show that CVCF is the best method when the level of noise is increased.