Predicting the Characteristics of High-Speed Serial Links Based on a Deep Neural Network (DNN)—Transformer Cascaded Model

Abstract

The design level of channel physical characteristics has a crucial influence on the transmission quality of high-speed serial links. However, channel design requires a complex simulation and verification process. In this paper, a cascade neural network model constructed of a Deep Neural Network (DNN) and a Transformer is proposed. This model takes physical features as inputs and imports a Single-Bit Response (SBR) as a connection, which is enhanced through predicting frequency characteristics and equalizer parameters. At the same time, signal integrity (SI) analysis and link optimization are achieved by predicting eye diagrams and channel operating margins (COMs). Additionally, Bayesian optimization based on the Gaussian process (GP) is employed for hyperparameter optimization (HPO). The results show that the DNN–Transformer cascaded model achieves high-precision predictions of multiple metrics in performance prediction and optimization, and the maximum relative error of the test-set results is less than 2% under the equalizer architecture of a 3-taps TX FFE, an RX CTLE with dual DC gain, and a 12-taps RX DFE, which is more powerful than other deep learning models in terms of prediction ability.

1. Introduction

As the transmission bandwidth of wireline serial link technology reaches the GHz level, it is no longer possible to ensure efficient signal transmission by simply optimizing the dielectric and layout structure. High-speed serial link systems suffer from serious signal integrity (SI) problems due to the skin effect, dielectric loss, crosstalk, reflections, and jitter; therefore, the SI analysis becomes more and more strict in the design stage of high-speed serial links. The simulation analysis of SI usually consists of two steps: electromagnetic field solvers (EMFSs) and circuit system simulation [1]. Firstly, an EMFS is used to obtain S-parameters to characterize the frequency response of the circuit. Then, these S-parameters are imported into the model circuit system for time-domain simulation to obtain the main SI metrics, including an eye diagram, the impulse response, and transient waveforms.

Although the traditional SI analysis based on physical models of high-speed links can offer high accuracy, it consumes a lot of time and computer resources. Input/Output Buffer Information Specification Algorithmic Model Interface (IBIS-AMI) is a behavioral model that simulates the input/output behavior and algorithms of end-to-end high-speed serial links, simplifying the internal physical details [2]. Compared to physical models, it has the advantages of speed, simplicity, and low resource consumption, but it has poor accuracy and lacks flexibility. Compliance standard template comparison is another commonly used method [3]. This can swiftly and intuitively estimate channel performance, but it inevitably discards the channel margins due to the need to strictly satisfy discrete metrics [4]. On the contrary, the channel operating margin (COM) technique can effectively overcome this disadvantage by searching the optimal design space of the whole link in the form of a signal-to-noise ratio (SNR). Compared to the traditional SI metrics such as eye diagrams and the bit error rate (BER), the COM approach shows advantages such as simpler operation, faster speed, and more efficient testing. Obviously, compared with Annex69B channel evaluation for 10 Gb/s Ethernet (10 GbE), using COMs is more accurate [5]. Although the COM approach can provide an effective way to evaluate and optimize high-speed links, it requires numerous iterations for spatial search and is not flexible enough to analyze high-capacity channels.

Machine learning (ML) has been widely used in the simulation and design of high-speed links in recent years, and shows the ability to improve the efficiency of SI analysis. The authors of [4,6,7,8,9,10] searched for features from simulation data and trained Artificial Neural Network (ANN), Deep Neural Network (DNN), and Least-Squares Support Vector Machine (LS-SVM) models to replace circuit system simulations for the accurate prediction of time-domain (TD) and frequency-domain metrics such as eye height (EH)/eye width (EW) and return loss (RL)/insertion loss (IL); however, the acquisition of simulation data consumes a lot of computational resources and time. Feedforward Neural Networks (FNNs), Random Forest Regression (RFR), and Support Vector Machines (SVMs) are used to achieve simple predictions of simulation data [11,12,13], such as for predicting S-parameters and impulse responses. When dealing with more complex simulation data, traditional ML methods may lose many features. The Recurrent Neural Network (RNN) is an ML model that can effectively capture the features of complex data, especially for sequence data such as S-parameters and impulse responses. The authors of [14,15,16,17,18,19,20] employ RNN and Long Short-Term Memory (LSTM) architecture to create surrogate models for predicting the transient response waveforms of complex high-speed links. These independent surrogate models can perform SI analysis based on simulation data, but cannot deal with the physical parameters of links. SI analysis based on physical parameters requires more complicated ML methods, and the authors of [21,22,23,24] achieved effective predictions of physical parameters used for assessing high-speed link performance by combining multiple deep learning algorithms. Moreover, balanced architecture optimization for high-speed serial links can be achieved based on ML [4,25,26,27]. In conclusion, performance prediction and architecture optimization [28] are two important parts of ML applications for high-speed serial links.

In this paper, a DNN–Transformer cascaded model is proposed for the SI analysis and optimization of high-speed serial links. This model can skip the use of EMFSs and circuit system simulation, and directly predicts SI metrics including EH/EW, IL/RL, impulse response, and COM values according to physical parameters. Meanwhile, the optimization of links can be achieved by using this model to predict the corresponding COM values and equalizer parameters for the links with different equalizer configurations. Furthermore, Bayesian optimization based on the Gaussian process (GP) is used to optimize the hyperparameters under the same conditions for different combinations of models. Compared with the prediction of high-speed link performance using traditional ML [6,7,8,9,10], DNN–Transformer can directly use the physical parameters of the link for analysis, and its prediction accuracy is significantly better. For the prediction of simulation data such as impulse responses, DNN–Transformer can capture more features and its prediction ability is more accurate than that achieved when using an RNN [14] or LSTM [20] model alone to create a surrogate model. The results show that the DNN–Transformer model can achieve more effective prediction. The performance and feasibility of the proposed method is illustrated by the prediction data and graphical results given in this paper.

The main contributions of this paper can be summarized as follows:

(1)

Based on the key physical parameters of channels, neural network models are used to directly analyze link performance, and this obviates the time-consuming processes of EMF solving and circuit system simulation.

(2)

Through neural network models, the accurate prediction of multiple SI indicators and equalizer parameters is achieved. Additionally, we show that SI analysis and link optimization can also be rapidly achieved.

(3)

A DNN–Transformer cascaded model is proposed, Bayesian optimization is used to tune the hyperparameters of this model, and its superior performance is demonstrated by comparing it with other models.

The rest of this paper is organized as follows: Section 2 discusses the basic principles of this work, including the principles of the SI and COM methods. Section 3 describes how the simulation dataset was created. In Section 4, the design idea of the DNN–Transformer model and test metrics are presented. Section 5 gives the numerical results to demonstrate the prediction performance of our model. Finally, Section 6 concludes the paper.

2. Fundamental Methods

2.1. Signal Integrity and S-Parameters

As transmission rates increase, the inefficiencies of traditional parallel signal transmission methods are becoming apparent and high-speed serial digital transmission is becoming widely used. Figure 1 shows the widely used SerDes structure in a high-speed serial link system. It comprises a transmitter chip with a serializer and a feedforward equalizer (FFE), a channel, and a receiver chip with a decision feedback equalizer (DFE) and a de-serializer. The FFE at the sending end mainly performs pre-emphasis on the signal to improve its high-frequency component. The composition of a wireline channel is complex, usually including traces, vias and connectors, etc. These physical structures exhibit low-pass filter characteristics and can enlarge the high-frequency loss of a signal, severely worsening the signal transmission quality and leading to SI problems. The continuous-time linear equalizer (CTLE) at the receiving end is mainly used to compensate for the high-frequency loss of the channel, eliminate the pre-cursors, and suppress the trailing of the pulse response. The DFE is mainly used to eliminate the post-cursors of the pulse response. The combination and parameter settings of the FFE, CTLE, and DFE have a significant impact on the SI of ultra-high-speed wireline serial communication links.

The main goal of SI analysis is to detect and reduce the factors that cause losses, such as dithering, reflection, and crosstalk. On the one hand, the SI can be analyzed from the frequency domain, as shown in Figure 2a. This mainly includes IL, RL, insertion loss deviation (ILD), and the insertion loss-to-crosstalk ratio (ICR), which are subsequently placed into a template for evaluation. On the other hand, the SI can be evaluated in the time domain, such as through eye diagrams, transient simulations, bathtub curves, and bit error rates, as shown in Figure 2b.

The critical method of traditional SI analysis involves the acquisition of S-parameters. These parameters contain comprehensive Frequency Domain (FD) characteristics of the transmission channel and offer a large amount of information on aspects such as reflection, crosstalk, and loss. Moreover, S-parameters can be employed in time-domain simulations to generate data such as eye diagrams and bathtub curves.

2.2. Channel Operating Margin

The COM method is a high-speed serial link characterization method recommended by the IEEE 802.3 working group for channel compliance testing. The official definition states that a “COM is a figure of merit (FOM) for channels determined from a minimum reference PHY architecture and channel s-parameters” [29]. This approach provides a relatively accurate and fair environment for the physical design of channels by considering various factors such as loss, reflection, inter-symbol interference (ISI), dispersion ISI, crosstalk, and device specifications, enabling a relatively accurate assessment of channel performance and the impact of aggressor channels on victim channels. The impact of equalizers has been considered in subsequent versions of the COM method, and the value of a COM based on the FOM can be improved by selecting equalizer parameter settings. Consequently, calculating the COM can also determine whether the channel quality meets the transceiver’s SI requirements [30]. As shown in Equation (1), the COM can be expressed by the ratio of available signal amplitude $A_s$ to statistical noise amplitude $A_n$.

$$\mathrm{COM}=20\times {\log }_{10}\left({\displaystyle \frac{A_s}{A_n}}\right)$$

(1)

The process of deriving the COM involves several essential steps, including determining the transfer function, converting the transfer function into the impulse response, applying transmitter and receiver equalizer algorithms, and performing statistical noise calculations. Figure 3 presents a flowchart of COM model derivation and depicts two paths for the victim and aggressor channels. Additionally, the model also considers transmitter and receiver package interference S (tp/rp), termination resistance Rd, equalizers, and filters. To better simulate actual channels, the COM model incorporates Gaussian white noise and jitter at the receiver end. Figure 4 shows the detailed process of COM value calculation.

The equalizer parameters are mainly set based on the channel loss. The FOM serves as a quantitative metric for assessing channel quality and equalizer performance. It takes into account several factors affecting SI, including ISI, jitter, crosstalk, and noise. The FOM calculation formula can be expressed as follows:

$$\mathrm{FOM}=10\times {\log }_{10}\left({\displaystyle \frac{A_s^2}{{\sigma }_{\mathit{TX}}^2+{\sigma }_{\mathit{ISI}}^2+{\sigma }_J^2+{\sigma }_{\mathit{XTK}}^2+{\sigma }_N^2}}\right)$$

(2)

The numerator $A_s$ is derived from the amplitude of the impulse response at $t_s$, which corresponds to the main cursor of the impulse response $h(t)$. The denominator includes the sum of variances from all noise, jitter, and interference components. ${\sigma }_{TX}^2$ represents the noise variance at the transmitter, ${\sigma }_{ISI}^2$ denotes the variance in the residual ISI amplitude, and ${\sigma }_J^2$ indicates the variance in jitter amplitude. In COM analysis, jitter is accounted for by converting horizontal jitter into vertical noise at the sampling instant $t_s$. Additionally, ${\sigma }_{XTK}^2$ represents the total crosstalk variance from all interference paths, while ${\sigma }_N^2$ denotes the Gaussian white noise at the receiver sampling point.

The COM approach entails exploring all possible parameter combinations of the TX and RX equalizers within a set range to find the configuration that maximizes the FOM. This process determines the values of $A_s$ and $A_n$ that result in the optimal FOM.

Traditional methods rely on various indicators such as jitter, eye height, and eye width, but the COM serves as a comprehensive metric for evaluating a serial link. It can significantly reduce the computation time and number of iterations, and provides an accurate evaluation of channel performance both before and after equalization.

3. Dataset Construction

3.1. Channel Design and Dataset Splitting

In this work, the channel datasets were generated using professional PCB design software. As shown in Figure 5, Altium Designer 20.0.13 and Allegro 17.4 (EDA software) were primarily utilized to create the layouts for high-speed differential lines. Subsequently, the PCB layout files were imported into Advanced Design System 2021 (ADS) or HFSS 2023 R1 for electromagnetic (EM) simulation. Our design conforms to the specifications for USB end-to-end differential transmission lines, including parameters such as line width and length, the characteristic impedance of transmission channels, and differential line spacing.

The differentiated characteristics of channel data, as presented in

Table 1. Channel feature parameters and quantitative methods.

Feature Variable	Input	Output
Er	PCB sheet permittivity real part and TanD	Eye diagrams COM values Equalizer parameters
Df	PCB sheet dissipation factor
Trace	PCB trace length
	PCB trace width
	Differential line space
	PCB trace thickness
	PCB trace conductivity
	PCB trace structure (profile)
Zd	Differential impedance
PCB Type	Stripline is 1 and microstrip is 0

, primarily include parameters such as material, trace dimensions, and PCB type. The board characteristics encompass both numerical features—such as permittivity (Er), the dissipation factor (Df), and board thickness—and categorical features such as the PCB type. Er consists of the sheet permittivity’s real part and the dielectric loss tangent angle (TanD). Df is derived from the Djordjevic model, which is defined by parameters including the low and high frequencies of TanD and relative high-frequency permittivity. In this study, Df is quantified according to the numerical values of TanD and its corresponding high and low frequencies. The PCB type is categorized into either the stripline type, represented by a numerical value of 1, or the microstrip type, represented by 0.

These trace features are quantified based on specific parameters such as the trace length, width, thickness, conductivity, profile, and differential line spacing. The profile is characterized by the geometric shape of the trace, such as the number of corners. The other characteristics are directly represented by their corresponding numerical values.

3.2. COM Configuration and Equalizer Setting

This work utilizes the COM 4.0 version program provided by IEEE 802.3. The configuration references the enhanced COM (eCOM) simulation setup with the USB4 Gen4 standard and incorporates the elements of the 50 GBASE-KR standard. The PAM4 configuration settings, as shown in

Table 2. Configuration of COM parameters.

Parameter	Symbol	Setting	Unit
Symbol rate	$f_b$	20	GBd
Number of signal levels	L	4
Samples per UI	M	32
Target detector error ratio	$DE{R}_0$	$10^{-8}$
Transmitter output voltage, victim	$A_v$	0.4	V
CTLE DC gain	$g_{DC}$	[−12:1:0]	dB
CTLE DC gain2	$g_{DC\_HP}$	[−6:1:0]	dB
CTLE HP pole	$f_{HP\_PZ}$	0.25	GHz
CTLE zero	$f_Z$	$f_b/2.5$	GHz
CTLE pole1	$f_{P1}$	$f_b/2.5$	GHz
CTLE pole2	$f_{P2}$	$f_b$
FFE main cursor	c (0)	0.62
FFE pre-cursor	c (−1)	[−0.18:0.02:0]
FFE post-cursor	c (1)	[−0.38:0.02:0]
DFE length	$N_b$	12	UI
DFE magnitude limit	$b_{\max }(1)$	0.75
	$b_{\max }(2~N_b)$	0.2
COM pass threshold	$th$	3	dB

, employ a combination of a transmitter FFE and receiver-side CTLE and DFE. This configuration includes the tap coefficients of the TX FFE and RX DFE, the adaptive range of DC gain, and the zero-pole positions of the RX CTLE. Additional parameters include the number of signal levels, denoted by L. For PAM4 signals, L is set to 4. According to the recommended PAM4 setup based on USB4, the symbol rate is set to 20 GBd, and the differential peak voltage output of the transmitter is set to 0.4 V, denoted as A_v. The USB4 Gen4 standard suggests that PAM3 should be used as the modulation method. The COM settings for PAM3 are roughly same as those of PAM4, except that the symbol rate is set to 25.6 GBd, and L is set to 3.

Based on the adopted configuration of TX FFE + RX CTLE + RX DFE, the prediction parameters include tap coefficients of the FFE, DC gain of the CTLE, and tap coefficients of the DFE. The TX FFE and RX CTLE data are obtained by searching for FOM values throughout the design space, while the DFE coefficients are determined according to the pulse response h(t) corresponding to the optimal FOM. The SI is significantly impacted by post-cursors closer to the main cursor, while the more distant post-cursors exert minimal influence, so only the first four post-cursors of the DFE are predicted.

3.3. Dataset Splitting

For this work, a total of 325 channels were collected. In this dataset, 280 channels were used to create an eye diagram dataset, referred to as Dataset A, and 215 channels were used to create a dataset for the COMs and predicate the parameters of the equalizers, referred to as Dataset B. Due to the impact of the FOM, the COM approach uses different equalizer parameter optimization methods for stripline and microstrip systems, which results in a weakened fitting ability of the ML model. Consequently, we standardized Dataset B using strip-line channels. Among the 215 channels in Dataset B, 170 channels are stripline channels in Dataset A, while 45 channels are newly created stripline channels. According to standard methodology, both datasets are partitioned into training, validation, and test sets. Part of the test set in Dataset B is extracted from the validation set. The split settings are detailed in

Table 3. Dataset split settings.

	Number of Channels
Datasets	A: 280	B: 215 (170 from A)
Training Set	184	140
Validation Set	40	40
Testing Set	56	40 (5 from the validation set)
Total	325

.

4. Construction of the Cascaded Model and Training

4.1. DNN

The M-P model, proposed by psychologist McCulloch and logician Pitts, is a mathematical model that was developed through an analysis and synthesis of the basic properties of neurons [31]. This model is crucial for the implementation of neural networks and constitutes the foundational unit of a DNN. The structure of this model is depicted in Figure 6. The $\mathit{x}={(x_1,x_2,\cdots ,x_n)}^T$ represents the n input features of the neuron, and ${\mathit{\omega }}_i=({\omega }_{i1},{\omega }_{i2},\cdots ,{\omega }_{in})$ denotes the weight vector responsible for linear weighted connections. The weight vector ω is optimized within neural networks to enhance the accuracy of predictions. The linear weighted output Z can be expressed as follows:

$$Z_i={\displaystyle \sum _{j=1}^n{\omega }_{ij}{x}_j}$$

(3)

Figure 6 depicts the structure of the M-P neuron model. The difference between Z and an offset θ is used as the input to the function $f(\cdot )$. The function $f(\cdot )$ is referred to as the activation function and used to derive the output of the neuron. θ can be considered as the weight of input $x_0$ with a fixed value of −1 and is also an optimization target of the neural network. In this paper, the ReLU function is adopted as the activation function because it can more effectively mitigate the occurrence of overfitting. The formula for this is shown in Equation (4).

$$f_{\mathrm{ReLU}}(x)=\max (0,x)$$

(4)

The output of the activation function is typically used as the input for adjacent layers. DNNs can complete tasks such as regression, classification, and recognition. DNNs are widely used for regression tasks in production and performance evaluation [4,6,7,8,9,32]. The DNN used in this paper, as shown in Figure 7, adopts a standard structure with multiple hidden layers, along with an input layer and a linear regression output layer [33], and employs a fully connected network architecture between its neurons.

In order to make the model converge better, the training model usually requires a cost function to evaluate the gap between model-predicted values and actual values. In this paper, Smooth L1 [34] is adopted as the cost function:

$${\mathrm{smooth}}_{\mathrm{L}1}=\{\begin{array}{ll}0.5\times {(y_n-{\widehat{y}}_n)}^2& \mathrm{if}\left|y_n-{\widehat{y}}_n\right|<1\\ \left|y_n-{\widehat{y}}_n\right|-0.5& \mathrm{otherwise}\end{array},$$

(5)

where $y_n$ and ${\widehat{y}}_n$ represent the real value and predicted value, respectively.

The performance of the model is evaluated using the cost function; then, an optimizer is employed to backpropagate and continuously minimize the cost function. The model parameters and weights are continuously updated until the expected performance is achieved. After comparing several different optimizers, the Adam optimizer was ultimately selected for use in this paper [35].

4.2. Transformer for Regression

The accuracy of equalizer parameters and COM values directly predicted by channel feature parameters is relatively low. In this work, the Single-Bit Response (SBR) before and after equalization is first predicted based on the feature parameters of the channel, and then the equalizer parameters and COM values are determined according to the predicted pulse response. The pulse response needs to be used as a time series for prediction. Traditionally, RNN or LSTM models are used for time-series prediction, but these networks have certain limitations. The Transformer model can effectively improve the gradient explosion problem that occurs in LSTM by using attention mechanisms, thereby enhancing prediction precision [36].

The formula of an attention mechanism is shown in Equation (6). It mainly consists of three inputs: Q (Query), K (Key), and V (Value), along with a softmax module. When Q, K, and V are identical, it is referred to as a self-attention mechanism.

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}})V$$

(6)

The complete Transformer architecture primarily includes an embedding layer, a positional encoding layer, an encoder layer, a decoder layer, and an output layer. In this work, since our task is a time-series prediction task rather than an NLP task, the positional encoding layer has been removed. The essence of time-series prediction can be viewed as a regression task; the model only needs to generate a single output value corresponding to each time point and does not need a decoder to generate sequence outputs. Using only the encoder layer is sufficient, which simplifies the model structure and reduces the computational load. As shown in Figure 8, our final design keeps the embedding layer, the encoder layer, and the final output layer. Additionally, in order to achieve faster convergence, the Gaussian Error Linear Unit (GELU) was chosen as the activation function.

$$f_{\mathrm{GELU}}(x)=xP(X\le x)=x\times {\displaystyle \frac{1}{2}}[1+\mathrm{erf}(x/\sqrt{2})]$$

(7)

4.3. Cascaded Model and Training

As illustrated in Figure 9, a cascaded Transformer and DNN model is utilized in this work to accurately predict the COM value and its corresponding equalizer coefficients. A total of 325 channel features have been collected as a dataset, and a DNN model is used to predict the spectral feature for each channel, specifically the IL and RL at the Nyquist frequency points. These predicted spectral data, along with the channel features, are then fed into a Transformer model to derive the pre-equalization SBR. Utilizing the pre-equalization SBR, the corresponding equalizer parameters under the COM configuration are calculated by the DNN model. In order to compute the post-equalization SBR, equalizer parameters combined with the channel features and frequency-domain features are then sent into another Transformer model. Finally, the sampled post-equalization SBR, channel features, and spectral features are used to compute the final corresponding COM value in the DNN model.

The input vector of physical characteristics for each channel can be defined as follows:

$$\mathit{X}={\left\{x_{ij}\right\}}_{i=1:N_C,j=1:N_P},$$

(8)

where $N_C$ represents the number of channels in the dataset, $N_P$ is the total number of channel features, and X is an $N_C\times N_P$ matrix. The element $x_{ij}$ denotes the quantified value of a specific physical feature of a channel: for instance, $x_{11}$ corresponds to the length feature value of first channel.

The DNN used in the cascaded model can be represented by Equation (9):

$$\begin{array}{ll}\mathit{Y}& =F_{DNN}(\mathit{Z},\mathit{\theta })\\ & ={\mathit{W}}^{(l)}{f}_{\mathrm{ReLU}}(\cdots ({\mathit{W}}^{(2)}{f}_{\mathrm{ReLU}}({\mathit{W}}^{(1)}\mathit{X}+{\theta }^{(1)})\\ & +{\theta }^{(2)})\cdots )+{\theta }^{(l)}\end{array},$$

(9)

where Z represents the linearly weighted output of input matrix X and model weights W, as described in Equation (3), and θ represents the offset. The index l denotes the layer of the DNN model, which includes the input layer, hidden layers, and output layer. Our network employs a total of four different DNN models with varying hyperparameters: ${\mathit{Y}}^f$ for predicting spectral features, ${\mathit{Y}}^e$ for predicting equalizer parameters, ${\mathit{Y}}^E$ for predicting the eye diagram, and ${\mathit{Y}}^C$ for predicting the COM.

The augmented matrix ${\mathit{X}}_f$ is obtained by horizontally concatenating the input matrix X of the first DNN model with its output ${\mathit{Y}}^f$:

$$\begin{array}{ll}{\mathit{X}}^f& =[\mathit{X}|{\mathit{Y}}^f]\\ & =\left[\begin{array}{cccccc}{x}_{11}& x_{12}& \cdots & x_{1N_P}& y_{1I}& y_{1R}\\ x_{21}& x_{22}& \cdots & x_{2N_P}& y_{2I}& {\mathrm{y}}_{2R}\\ ⋮& ⋮& \ddots & ⋮& ⋮& ⋮\\ x_{N_{C}1}& x_{N_{C}2}& \cdots & x_{N_C{N}_P}& y_{N_{C}I}& y_{N_{C}R}\end{array}\right]\end{array},$$

(10)

where the column vector ${\mathit{y}}_I={(y_{1I},y_{2I},\cdots ,y_{N_{c}I})}^T$ represents the predicted channel IL and ${\mathit{y}}_R={(y_{1R},y_{2R},\cdots ,y_{N_{c}R})}^T$ represents the predicted channel RL.

To enhance the accuracy of the predicted COM value, the pulse response is extracted from the pre- and post-equalization SBR in the COM tool. The peak corresponding time point is taken as the main cursor, and the amplitudes of four pre-cursors and eight post-cursors determined by the COM sampling interval are selected as the required SBR. Then, the pulse of each channel is flattened and the corresponding time labels are inserted, forming a sequential feature vector. Equation (11) represents the sequence of amplitude values. After this processing, the Transformer model can be employed to accurately predict the SBR.

$$\{\begin{array}{l}{SBR}_1(t)=F_{Trans}[x_{11}(t),x_{12}(t),\cdots ,\\ x_{1N_P}(t),y_{1I}(t),y_{1R}(t)]\\ {\mathit{S}\mathit{B}\mathit{R}}_1=[SB{R}_1(t-t_0),\cdots ,\\ SB{R}_1(t),\cdots ,SB{R}_1(t+t_1)]\end{array},$$

(11)

where $F_{Trans}$ denotes the Transformer model being utilized, $SB{R}_1(t)$ represents the SBR corresponding to the input feature vector of the channel at a specific time point, and ${\mathit{S}\mathit{B}\mathit{R}}_1$ represents the SBR sequence of the channel. $t_0$ and $t_1$ are the left and right boundaries of the time range, respectively.

By feeding the SBR into the subsequent DNN model with distinct parameters, the prediction of equalizer parameters can be achieved via Equation (12):

$${\mathit{Y}}^e=F_{DNN}({\mathit{S}\mathit{B}\mathit{R}}_1,{\mathit{S}\mathit{B}\mathit{R}}_2,\cdots ,{\mathit{S}\mathit{B}\mathit{R}}_{N_C})$$

(12)

By integrating the equalizer parameters with the input featuring spectral characteristics, an augmented matrix $SB{R}_{N_C}^{eq}(t)$ is constructed for the prediction of post-equalization SBR. This matrix, via Equation (13), is then processed into a time-dependent sequence following the methodology presented in Equation (11).

$$\{\begin{array}{l}SB{R}_{N_C}^{eq}(t)=F_{Trans}[x_{N_{C}1}(t),\cdots ,x_{{\mathrm{N}}_{\mathrm{C}}{\mathrm{N}}_{\mathrm{P}}}(t),\\ {\mathrm{y}}_{{\mathrm{N}}_{\mathrm{C}}\mathrm{I}}(t),{\mathrm{y}}_{{\mathrm{N}}_{\mathrm{C}}\mathrm{R}}(t),{\mathit{Y}}_{{\mathrm{N}}_{\mathrm{C}}}^{\mathrm{e}}(t)]\\ \begin{array}{l}{\mathit{S}\mathit{B}\mathit{R}}_{N_C}^{eq}=[SB{R}_{N_C}^{eq}(t-t_0),\cdots ,\\ SB{R}_{N_C}^{eq}(t),\cdots ,SB{R}_{N_C}^{eq}(t+t_1)]\\ {\mathit{Y}}_{N_C}^e(t)=[{\mathit{y}}_{FFE-N_C}^{Taps}(t),{\mathit{y}}_{CTLE-N_C}^{DCgain}(t),{\mathit{y}}_{DFE-N_C}^{Taps}(t)]\end{array}\end{array}$$

(13)

Finally, the obtained post-equalization SBR is concatenated with ${\mathit{X}}^f$ to form the final input features for predicting the COM value.

Before formally training the cascaded model, we standardized the input data and employed a pre-training strategy. Initially, the DNN models $F_{DNN}^{freq}(\mathit{X})$ and $F_{DNN}^{eq}(\mathit{X})$ designed for predicting spectral features and equalizer parameters, respectively, were trained independently, and their corresponding parameters were saved. These pre-trained DNN models were then integrated with the necessary Transformer model to construct the complete cascaded model for predicting the equalizer parameters and COM, as depicted in Figure 9. For training the cascaded model, Smooth L1 was used as the cost function and the Adam algorithm was applied to update the model parameters. With the help of the optuna toolkit, model structuring and hyperparameter optimization (HPO) were performed via Bayesian optimization using the GP [37,38]. The chosen acquisition function was the Expected Improvement (EI) function, with an initial set of 10 observation points, allowing for effective HPO within a computationally feasible scope.

Compared to the global grid search, random grid search, and halving search methods, Bayesian optimization is more efficient for HPO and requires less optimization time. Additionally, the K-Fold cross-validation method is employed in this paper to enhance the reliability of the model evaluation.

After completing the model’s training, its predictive capabilities can be assessed with several different metrics. One of these metrics, shown in Equation (14), is the RMSE. The squaring operation in RMSE increases its sensitivity to errors, making it particularly responsive to outliers. This increased sensitivity can also make it overly responsive to outliers, but the square-root operation cannot intuitively reflect the actual magnitude of error.

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n{(y_n-{\widehat{y}}_n)}^2}}$$

(14)

MAE is calculated according to the average value of absolute differences between the predicted and actual values. Compared to RMSE, MAE is less sensitive to outliers and provides a more intuitive reflection of the actual magnitude of errors. However, this can lead to an underestimation of the impact of the errors.

$$\mathrm{MAE}={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n\left|y_n-{\widehat{y}}_n\right|}$$

(15)

MAPE is the average value of absolute percentage differences between the predicted and actual values. Compared to RMSE and MAE, MAPE is easier to understand and allows for the comparison of prediction results across different scales. It is more suitable for situations with significant changes in actual values. However, when the actual values are close to zero, the error can become large, so MAPE is unsuitable for datasets containing zero or near-zero values.

$$\mathrm{MAPE}={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n\left|\frac{y_n-{\widehat{y}}_n}{y_n}\right|}$$

(16)

5. Numerical Results

In this work, the neural network model has been constructed using a Python library based on Pytorch 2.0.1. The computer used for model computation included a GeForce RTX 4070 and an Intel Core i5-12600 KF 3.7 GHz.

5.1. IL and RL

As shown in Figure 10, the IL and RL are losses at Nyquist frequency, and the model’s predictions of IL are more accurate than its predictions of RL. This is because IL is primarily influenced by the dielectric loss and input features, including dielectric loss-related parameters. Conversely, RL is affected by additional factors such as impedance mismatches and complex trace shapes, complicating its prediction. Therefore, the accuracy of IL prediction is better than that of RL prediction. The relative error of RL prediction is 17.4%, but this is sufficient to support the accurate prediction of eye diagrams and COM values.

5.2. EH/EW of the Eye Diagrams

Figure 11 shows the relative errors in eye height and width for 40 test channels with the PAM3 and PAM4 modulation methods. Channels No. 1 to No. 32 are stripline channels, while channels No. 33 to No. 56 are microstrip channels. Channels No. 1 to No. 10 exhibit more complex trace paths, resulting in larger errors in eye height and eye width. In contrast, channels No. 11 to No. 23, which have simpler routing with straighter or fewer bend paths, show better fitting precision. Most channels in the dataset utilize FR-4 as the dielectric layer, whereas channels No. 24 to No. 32 employ alternative materials, resulting in relatively larger errors. Compared to the stripline channels, channels No. 33 to No. 56, which utilize microstrip lines, show significantly higher relative errors, demonstrating that microstrip lines have weaker anti-interference capabilities than striplines.

Table 4. Prediction of eye diagrams with the DNN model.

Eye	Modulation	RMSE	MAE	MAPE	MRE (%)
EH (mV)	PAM3	4.8 × 10−1	3.3 × 10−1	5.7 × 10−3	2.5
	PAM4	4.0 × 10−1	2.9 × 10−1	8.1 × 10−3	2.8
EW (UI)	PAM3	4.4 × 10−3	3.4 × 10−3	9.3 × 10−3	2.7
	PAM4	4.4 × 10−3	3.6 × 10−3	1.2 × 10−2	3.5

shows the RMSE, MAE, MAPE, and MRE for the eye diagram prediction with the PAM3 and PAM4 modulation methods. It is evident that the prediction results exhibit good convergence and precision. Because the EH unit is chosen as mV, the RMSE and MAE metrics increase by an order of magnitude compared to EW. The results in this table, combined with those in Figure 11, demonstrate that the PAM3 and PAM4 models are effective and exhibit high accuracy.

5.3. COM

Because many pre-cursor and post-cursor SBR values are close or equal to zero, MAPE and MRE are not suitable for evaluating the accuracy of SBR predictions, and only RMSE and MAE are used to evaluate the prediction accuracy for the SBR. For the SBR before equalization of the 40 test channels, the RMSE is 7.3 × 10⁻⁴ and the MAE is 6.3 × 10⁻⁴. For the SBR after equalization, the RMSE is 1.1 × 10⁻³ and the MAE is 1.6 × 10⁻⁴. Below, we use a randomly selected channel (ID: 3) from the test set to illustrate the SBR prediction performance. As depicted in Figure 12, both the pre-equalization and post-equalization SBR can be predicted by the Transformer model with high precision.

For the PAM4 signal format, Figure 13 provides a concise demonstration. We selected an equalizer configuration that includes a 3-taps TX FFE, an RX CTLE with dual DC gain, and a 12-taps RX DFE (hereafter referred to as the standard equalizer configuration). As shown in Figure 13, channels No. 7 and No. 30 have large relative errors due to their complex shapes. Some channels with high RL, such as channels No. 7–No. 9, also exhibit significant relative error. The relative error of the predicted COM values for each test channel is within 2%, demonstrating the ability of the DNN–Transformer model to achieve the required prediction accuracy.

Here, the influences of different combinations of equalizers on the prediction precision of COM values are analyzed deeply. As shown in

Table 5. Predicted COM error values for the cascaded model.

COM	RMSE	MAE	MAPE	MRE (%)
3-taps FFE + CTLE + 12-taps DFE	4.5 × 10−2	3.5 × 10−2	5.1 × 10−3	1.6%
3-taps FFE + CTLE + 8-taps DFE	5.0 × 10−2	3.7 × 10−2	5.6 × 10−3	2.0%
3-taps FFE + CTLE + 4-taps DFE	4.2 × 10−2	3.2 × 10−2	4.1 × 10−3	1.7%
CTLE + 12-taps DFE	9.6 × 10−2	7.1 × 10−2	1.4 × 10−2	5.7%
12-taps DFE	1.6 × 10−1	1.1 × 10−1	3.0 × 10−2	17.5%

, we configured five different equalizer combinations. The results indicate that our proposed cascaded model achieves the expected prediction performance across various equalizer combinations. However, it is apparent that a reduction in the variety of equalizers weakens the model’s generalization capability and decreases its prediction accuracy. When only the DFE was enabled, the RMSE increased to 1.6 × 10⁻¹, and the MRE reached 17.5%. We attribute this to the optimization method of this equalizer within the COM framework and the fact that the cascaded model also takes into account predictions for both the CTLE and FFE. Therefore, we adjusted the network structure by removing unused equalizers in the cascaded model for different equalizer combinations. This modification has resulted in an improvement in the model’s prediction accuracy for COM values, with the MRE decreasing from 17.5% to 11.5%.

Below, a test channel (ID: 3) is used to illustrate the prediction capability of the model with the first four taps of the DFE and three taps of the FFE. As shown in Figure 14a, the cascaded model realizes highly accurate predictions. Furthermore, the prediction results for the main DC gain of the CTLE for the 40 test channels, as presented in Figure 14b, also indicate a high level of accuracy.

The predictive capabilities of different cascaded model combinations were investigated as follows. The equalizer configuration was set as the standard configuration, and the HPO parameters were uniformly configured, with the learning rate search range set from 0.0001 to 0.01, conducting 20 trials for optimization. Given the characteristics of the SBR, we tested the predictive performance of standalone models, including a DNN, an LSTM network, and a Transformer. The DNN combined with two sequence prediction models, LSTM and a Transformer, was selected as a cascaded scheme. The DNN-LSTM model replaces the Transformer module in Figure 9 with a typical LSTM network. The DNN, LSTM, and Transformer use independent models without time label processing modules, so these models can only achieve eye diagram and COM value predictions. The best structural parameters of these models—such as the number of hidden layers and the hidden layer size of the LSTM, and the head number, vector dimension, and feedforward network dimension of the Transformer’s attention mechanism—were all obtained through Bayesian optimization. As presented in

Table 6. Predicted error values of different models.

Model	RMSE	MAE	MAPE	MRE (%)
DNN	6.9 × 10−2	4.3 × 10−2	7.8 × 10−3	6.5
LSTM	5.3 × 10−2	4.4 × 10−2	6.4 × 10−3	2.0
DNN-LSTM	5.0 × 10−2	4.1 × 10−2	6.0 × 10−3	1.8
Transformer	4.7 × 10−2	3.7 × 10−2	5.5 × 10−3	1.7
DNN–Transformer	4.6 × 10−2	3.5 × 10−2	5.1 × 10−3	1.6

, the cascaded models demonstrate superior predictive capabilities compared to the standalone models. Among the cascaded models, those incorporating a Transformer display the best predictive performance, indicating that the DNN–Transformer cascaded model is the optimal COM prediction model.

As shown in

Table 7. Functional comparison of different models for high-speed links.

Model	Physical Parameters Variable	Transient Waveform Prediction	FD Prediction	Link Optimization	GPU/CPU Time (ms)	RMSE	MAE
SVM [13]	Yes	No	No	No	<5 *	3.8 × 10−1	3.1 × 10−1
RFR [12]	Yes	No	Yes	No	<5 *	4.8 × 10−1	3.8 × 10−1
DNN [9]	Yes	No	No	No	<5	6.9 × 10−2	4.3 × 10−2
LSTM [20]	No	Yes	No	No	<5	5.3 × 10−2	4.4 × 10−2
GNN-RNN [21]	Yes	Yes	No	No	567.1	\	\
DNN–Transformer	Yes	Yes	Yes	Yes	792.8 *	4.6 × 10−2	3.5 × 10−2

* GPU inference time.

, we have compared the DNN–Transformer model with the models presented in the Introduction section in terms of functionality. The traditional LSTM [20] structure could not achieve prediction under physical parameter variables. Traditional machine learning models such as the SVM [13], RFR [12], and DNN [9] models have limited processing capabilities for sequences and thus cannot predict transient waveforms. The GNN-RNN [21] model simplifies the interconnected components and circuits, meaning that some of the actual parameters of links are lost and the performance is decreased. The above models could not achieve link optimization, and RFR [12] could only complete FD prediction. In contrast, the proposed DNN–Transformer is more practical. This model can effectively achieve these above behavioral tasks and performs better than other models in RMSE and MAE for COM value prediction. Due to the increased complexity of this model, its inference time is increased, but it still has a significant time advantage compared to traditional EM simulation whose single-solution time is several hours.

6. Conclusions

In this paper, a DNN–Transformer cascaded neural network is proposed for effectively analyzing the SI in high-speed serial links. During the dataset creation process, we referenced the USB4 Gen4 and 50GBASE-KR standards for PCB design and electromagnetic simulation, and used the physical design parameters of each channel as inputs for the model. This DNN–Transformer model is used in this paper to extract the features from physical design parameters of channels and successfully predict the eye diagram data and COM values of test links. In addition, this deep learning model can successfully predict the SBR before and after equalization, and main equalizer parameters for different combinations are also accurately predicted. For the model’s training, we employed a Bayesian optimization method based on the GP for HPO. Finally, this paper compares DNN–Transformer with various other models such as DNN, LSTM, Transformer, and DNN-LSTM models. The results shows that our DNN–Transformer cascaded model accurately achieves performance prediction and equalization architecture optimization for high-speed serial links, and the MRE in its COM prediction results for the test set, with an equalizer configuration comprising a 3-taps TX FFE, an RX CTLE with dual DC gain, and a 12-taps RX DFE, is 1.6%.