*2.3. Cost Function of DBN-1 and DBN-2 Equalizers*

#### 2.3.1. Cost Function of the Adaptive DBN-1-Based Equalizer

As we know, CMA is a typical blind equalizer for OOK or QPSK modulation format with a single reference radius. To reduce the large residual error after convergence, CMA blind equalization can be combined with a DBN-1 equalizer. The corresponding filter tap weight *hnn* and the cost function of the DBN-1 equalizer based on the CMA algorithm are defined as

$$h\_{nn} \to h\_{nn} + \mu e\_{n\\_CMA} \mathfrak{x}(n) \tag{5}$$

$$J\_{n\\_CMA} = \frac{1}{2} \sum\_{n=1}^{Q} \left( R\_1 - \left| O(n) \right| \right)^2 \tag{6}$$

where *μ* is defined as the CMA convergence parameter, *R*<sup>1</sup> is the constant module of signals, *O*(*n*) is the output of the DBN-1 network, and the input sequence *x*(*n*) in the optimization network is the test signal with a size of *Q.*

The adaptive error function can be defined as *en*\_*CMA* = *R*<sup>1</sup> − |*O*(*n*)|. In practical terms, we can find the optimum *μ* and tap number to obtain better channel equalization. Moreover, the weight value can be further optimized with the aid of adaptive equalization and the BP algorithm, which can be given as

$$w\_{ij}^{k+} = w\_{ij}^k - \Delta w\_{ij}^k = w\_{ij}^k - \eta \frac{\partial I\_{n \text{ \textquotedblleft}CMA}}{\partial w\_{ij}^k} \tag{7}$$

### 2.3.2. Cost Function of Blind DBN-2-Based Equalizer

Unlike forwarding propagation steps, the weight values and model hyper-parameters are updated based on the backpropagation (BP) algorithm. According to the minimum mean square error (MMSE) algorithm, the cost function of the DBN-2 blind equalizer based on the DD-LMS algorithm can be defined as

$$J\_{n\\_ddlms} = \frac{1}{2} \sum\_{n=1}^{S} \left( T\_n - O\_2(n) \right)^2 \tag{8}$$

Compared the obtained value *O*2(*n*) with the corresponding expected value *Tn*, the error *en* = *Tn* − *O*2(*n*) is sent to the network with a reverse training algorithm. Thus, the connected weight vector *wk*<sup>+</sup> *ij* can be iteratively updated until the desired epoch or error value is reached, which can be represented as

$$w\_{ij}^{k+} = w\_{ij}^k - \Delta w\_{ij}^k = w\_{ij}^k - \eta \frac{\partial I\_{n\text{\textquotedblleft}dlms}}{\partial w\_{ij}^k} \tag{9}$$

where *η* is the learning rate and Δ represents the gradient operation. In order to accurately update the weight value of every nonlinear node, we must calculate the gradient of the whole training sequence. Owing to the introduction of the blind error function, we will decrease the training size and improve the computation speed effectively.

Thus, our proposed J-DBN equalizer comprises two parts, including the DBN-1 adaptive equalization and the DBN-2 blind equalization. Note that the former updating of the weight value is based on the traditional BP algorithm, while the latter deploys the blind equalization algorithm to optimize the network further.

### **3. Experimental Setup**

In our previous work, we have established some real-time photonics-aided THz seamless integration transmission systems [12–15]. In order to verify the effectiveness of our proposed algorithms in this paper, we further perform an experimental demonstration for 144−Gbps photonics-assisted THz wireless transmission at 500 GHz enabled by J-DBN equalization. A detailed description of the experimental setup is shown in Figure 3, including the optical and THz transmitter modules, THz 2 × 2 MIMO wireless link, THz receiver module, and the off-line DSP blocks. For a fair comparison, there were four alternative algorithms included in the experiment at the off-line DSP blocks.
