Vibration Control with Reinforcement Learning Based on Multi-Reward Lightweight Networks

Abstract

This paper proposes a reinforcement learning method using a deep residual shrinkage network based on multi-reward priority experience playback for high-frequency and high-dimensional continuous vibration control. Firstly, we keep the underlying equipment unchanged and construct a vibration system simulator using FIR filters to ensure the complete fidelity of the physical model. Then, by interacting with the simulator using our proposed algorithm, we identify the optimal control strategy, which is directly applied to real-world scenarios in the form of a neural network. A multi-reward mechanism is proposed to assist the lightweight network to find a near-optimal control strategy, and a priority experience playback mechanism is used to prioritize the data to accelerate the convergence speed of the neural network and improve the data utilization efficiency. At the same time, the deep residual shrinkage network is introduced to realize adaptive denoising and lightweightness of the neural network. The experimental results indicate that under narrowband white-noise excitation ranging from 0 to 100 Hz, the DDPG algorithm achieved a vibration reduction effect of 12.728 dB, while our algorithm achieved a vibration reduction effect of 20.240 dB. Meanwhile, the network parameters were reduced by more than 7.5 times.

1. Introduction

When it comes to active vibration control, we face a high-dimensional, high-frequency, continuous, and complex dynamic problem. In order to cope with this challenge, a variety of traditional methods have been proposed. For example, PID controllers [1,2,3,4] are used to control complex objects, especially those with large time delays, rapid parameter changes, significant uncertainties, and severe nonlinearity. However, the PID controller has poor control quality in these cases, and a large number of parameters need to be designed in advance. Another common method employs the LQR controller [5,6,7], which uses a quadratic performance index. However, in practical applications, not every scenario is suitable for such an indicator, and it can only be applied to systems that can be represented by state-space equations. Another method involves the FxLMS controller [8,9,10], which not only needs to construct the reference signal, but also needs to identify the auxiliary path. However, studies have shown that although these traditional methods can play a role in control, they all require a lot of design work and manual intervention.

In recent years, the use of reinforcement learning (RL) technology to improve the efficiency of vibration controller designs has become a method of great concern. This method uses the reinforcement learning algorithm to automatically obtain the parameters of the controller, which brings new possibilities for vibration control. Compared with traditional methods, the reinforcement learning method is a data-driven approach, which does not require physical information on vibration control, such as the mass matrix, damping matrix, and stiffness matrix, thus reducing the required engineering and design investment. In fact, the use of reinforcement learning methods to construct agents to learn vibration control in complex environments has achieved remarkable results [11,12,13,14,15]. This method has been applied in many fields, such as for suspension vibration control [16,17], manipulator vibration control [18,19,20], magnetorheological damper vibration control [21,22], and other vibration control applications [23,24,25], and has made remarkable progress.

In the field of suspension vibration control [16,17], Li et al. [16] integrated model-free learning with model-based safety supervision. They used the underlying system dynamics and security-related constraints, used recursive feasibility techniques to construct security sets, and integrated them into the detection system constraints of reinforcement learning. This secure reinforcement learning framework was applied to the active suspension control system. Their experiments show that this secure reinforcement learning system can learn the control strategy safely and has better performance than the standard controller. Zhao et al. [17] discussed the set value tracking problem for the suspension system of a medium-speed maglev train, aiming at maintaining a fixed distance between the train and the track. They describe the set point tracking problem as a continuous-state and continuous-action Markov decision process under unknown transition probabilities, and obtain the state feedback controller from the sampled data of the suspension system through a Deterministic Policy Gradient and neural network approximation reinforcement learning algorithm. Their simulation results demonstrate the high performance of this model-free method, especially given that the response speed of their controller is 50 times faster than that of PID and no additional modeling work is required.

In the field of robotic arm control [18,19,20], Ding et al. [18] proposed an innovative human–computer interaction system and applied it to the cooperative operation control task of a robotic arm. Their human–computer interaction system includes an impedance model controller and a manipulator controller. The model-based reinforcement learning method is applied to the impedance mode for operator adaptation. The newly designed adaptive manipulator controller realizes a speed-free filter. Their simulation results verify the effectiveness of the human–computer interaction system based on the real manipulator parameters. On the other hand, Adel et al. [19] proposed a learning framework for robot manipulators based on nonlinear model predictive control, which aims to solve the motion planning problem of robot manipulators in the presence of obstacles. The cost function, system constraints, and manipulator model were parameterized by using the value function of the reinforcement learning framework and the myopic value of the action-value function. The controller was applied to the robot manipulator of a 6-DOF model. Their numerical simulation results show that this controller can effectively control the attitude of the end effector, avoid the collision between the manipulator and the obstacle, and improve the efficiency of the control loop by 21%. Additionally, Vighnesh et al. [20] designed a two-wheeled inverted pendulum mobile manipulator for retail shelf inspections. In comparison to PID control, in simulation tests, the controller based on reinforcement learning exhibited stronger robustness against variations in initial conditions, changes in inertia parameters, and disturbances applied to the robot.

In the field of magnetorheological damper vibration control [21,22], Park et al. [21] proposed a reinforcement learning method for magnetorheological elastomer vibration control based on Q-learning. Their research shows that it is feasible to use a model-free reinforcement learning model to realize an adaptive controller based on the application of highly nonlinear magnetorheological elastomers. When compared with a model without the controller, this method suppressed the vibration level by up to 57%. In addition, Yuan et al. [22] proposed a magnetorheological damping semi-active control strategy based on a reinforcement learning Q-learning algorithm. Their simulation results show that this method is simpler, easier to implement, and more robust. They used the reinforcement learning semi-active control strategy to study the vibration reduction of a two-story frame structure. The results show that the strategy is superior to the simple bang–bang control method.

At present, some progress has been made in the combination of reinforcement learning and finite-impulse response filters. Feng et al. [26] successfully solved the multi-input/multi-output actual vibration control problem in a frequency range by using RL and FIR filters, which marks the first time that reinforcement learning has been used to solve high-dimensional high-frequency vibration control problems. In order to meet the lightweight requirements of the underlying equipment, they designed the simplest fully connected neural network. However, this method may face the challenge of overfitting or underfitting when solving more complex and variable dynamic problems. In order to solve this problem, we try to find a way to both maintain lightweightness and solve complex problems.

In the field of deep learning, the way to achieve lightweightness is to reduce resource consumption and model complexity. In this regard, some achievements have been made [27,28,29,30]. Among the research on lightweightness, He et al. [27] proposed a residual network framework to reduce the burden of network training. They explicitly redefined each layer as the input of the reference layer and learned the residual function instead of learning an unknown function. Through comprehensive experimental data, they clarified that their residual network is easier to optimize, and that increasing the network depth can achieve higher accuracy. This method improved the results in the COCO target detection dataset by 28%, and won the first place in the COCO target detection and image segmentation tasks. On the other hand, Howard et al. [28] proposed an efficient model called MobileNets, which uses deep separable convolution to construct a lightweight deep neural network. Through extensive resource and accuracy experiments, MobileNets showed strong performance compared with other popular ImageNet classification models. They also demonstrate the effectiveness of this method in object detection, fine-grained classification, face attributes, and large-scale geolocation. Zhang et al. [29] proposed a computationally efficient CNN architecture called ShuffleNet. This method uses two new operations, point group convolution, and channel rearrangement, which greatly reduces the computational cost while maintaining accuracy. Their experiments on ImageNet classification and MS COCO target detection show that ShuffleNet is superior to other structures. On ARM-based mobile devices, the speedup of ShuffleNet is about 13 times higher than that of AlexNet, and it maintains considerable accuracy. In addition, Zhao et al. [30] proposed a deep residual shrinkage network to improve the feature learning ability of high-noise vibration signals. They inserted the adaptive soft threshold as a nonlinear transformation layer into the deep architecture to eliminate unimportant features. The effectiveness of their developed method was verified through various types of noise experiments.

At present, reinforcement learning forms a very mature system. It is mainly divided into two categories: value-based learning and probability-based learning, and has algorithms that combine these two methods. Probability-based reinforcement learning is one of the most direct methods and makes decisions by directly outputting the probability of the next possible action. In this method, each action has the opportunity to be selected, but the size of these opportunities is different. For example, the Policy Gradients approach [31] is a probability-based approach. In contrast, the value-based approach involves outputting the value of all actions and selecting the action with the highest value in the decision-making process. For example, algorithms such as Q-learning [32] are value-based methods. Combining the advantages of these two methods, some more optimized algorithms have been derived, such as Actor–Critic [33]. In this structure, the Actor makes an action based on probability, while the Critic assesses the value of the action. This combination optimizes the original Policy Gradients approach and accelerates the learning process. At present, the AC method has many extensions, including the Deep Deterministic Policy Gradient (DDPG) [34], Soft Actor–Critic (SAC) [35], and Twin Delayed Deep Deterministic Policy Gradient (TD3) [36] techniques.

Regarding the Actor–Critic method of reinforcement learning, Lillicrap et al. [34] proposed an Actor–Critic model-free algorithm based on Deterministic Policy Gradients, which can operate in a continuous-action space. They successfully solved the challenges of more than 20 simulated physical tasks by using the same learning algorithm, network architecture, and hyperparameters. Haarnoja et al. [35] described the Soft Actor–Critic approach, which is an offline strategy Actor–Critic algorithm based on a reinforcement learning framework. Its goal is to maximize both the expected return and entropy. This method is superior to the previous online strategy and offline strategy methods in terms of the sample utilization efficiency and progressive performance. In addition, Fujimoto et al. [36] found that in value-based reinforcement learning methods, known function approximation errors may lead to overestimation and suboptimal strategies for value estimation. Based on this, they propose a new mechanism called Twin Delayed Deep Deterministic Policy Gradient to mitigate the overestimation issues of Actor–Critic algorithms. They suggest constraining overestimation by taking the minimum of a pair of critics based on the foundation of double Q-learning. When evaluating their approach, it consistently outperformed current state-of-the-art methods across various testing environments.

However, due to the large number of parameters required in stacking neural networks to solve complex problems, previous algorithms have limitations such as algorithmic simplicity or a larger parameter count, impacting the timely control of controllers. To address this issue, this paper proposes vibration control with reinforcement learning based on a multi-reward lightweight network. Specifically, it is a deep residual shrinkage lightweight method based on deep reinforcement learning with multi-reward prioritized experience replay (DRSL-MPER). We introduce a multi-reward mechanism to assist the TD3 algorithm in finding control policies close to optimal. At the same time, we also introduce a priority experience playback mechanism to assign priority to the data, thereby accelerating the convergence speed of the neural network and improving the data utilization efficiency. In addition, we also introduce a deep residual shrinkage network to achieve adaptive denoising and lightweightness of the neural network. The algorithm proposed in this paper has two significant and practical advantages. Firstly, it employs a lightweight network, which not only enhances the computational efficiency and reduces the energy consumption but also facilitates deployment and maintenance in real-world scenarios, while supporting rapid iterative updates of the system. Secondly, the algorithm demonstrates excellent control performance and good generalization capabilities, which can significantly improve the performance and lifespan of equipment, enhance the reliability and safety of the system, and increase its adaptability and flexibility.

The rest of this paper is organized as follows: Firstly, the vibration control problem is described. Secondly, our lightweight method of multi-reward priority experience playback based on deep reinforcement learning is proposed. Then, the vibration controller designed using this method is verified through experiments. Finally, we draw our conclusions.

2. Problem Formulation

2.1. Dynamic Model

Figure 1 shows the dynamic model of the high-dimensional high-frequency vibration control system considered in this paper. The vibration exciter $m$ is installed on the equipment platform $M$, which is supported on the foundation by four rubber dampers. The rubber dampers can be regarded as a spring damping system composed of linear spring $k$ and linear damping $c$. Four electromagnetic actuators are mounted on the equipment platform, which generate counteracting control forces to reduce the vibration of the equipment platform caused by the vibration exciter. The rotation angles of the equipment platform around its horizontal inertia principal axes paralleling to the $x$-axis and $y$-axis are $\psi$ and $\theta$. The vibration exciter generates excitation forces in three directions, denoted as $F_{ex},F_{ey},F_{ez}$. However, since the actuators can only generate control forces in the $z$-direction ($F_{c1},F_{c2},F_{c3},F_{c4}$), only the translational motion along the $z$-axis and the rotational motion around the horizontal axis are considered. Thus, there are three generalized coordinates in the dynamic model: $z,\psi ,\theta$.

Using the Lagrange equation, the dynamic equations of the vibration control system can be described as follows:

$$\begin{array}{cc}\hfill \left(M+m\right)\ddot{z}& + c\left[4\dot{z}+\left(4a-L_y\right)\dot{\psi }+\left(4b-L_x\right)\dot{\theta }\right]\hfill \\ & + k\left[4z+\left(4a-L_y\right)\psi +\left(4b-L_x\right)\theta \right]\hfill \\ & = -F_{ez}-F_{c1}-F_{c2}-F_{c3}-F_{c4}\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill J_x\ddot{\psi }+c[\left(4a-2L_y\right)\dot{z}& + \left(4a^2-L_{y}a+2L_y^2-2L_{y}a\right)\dot{\psi }\hfill \\ & + \left(4ab-L_{x}a+L_y{L}_x-2L_{y}b\right)\dot{\theta }]\hfill \\ & + k[\left(4a-2L_y\right)z+\left(4a^2-L_{y}a+2L_y^2-2L_{y}a\right)\psi \hfill \\ & + \left(4ab-L_{x}a+L_y{L}_x-2L_{y}b\right)\theta ]\hfill \\ & = F_{c1}\left(L_x-a\right)-F_{c2}a-F_{c3}a+F_{c4}\left(L_x-a\right)\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill J_y\ddot{\theta }+c[\left(4b-2L_x\right)\dot{z}& + \left(4ab-L_{y}b+L_x{L}_y-2L_{x}a\right)\dot{\psi }\hfill \\ & + \left(4b^2-L_{x}b+2L_x^2-2L_{x}b\right)\dot{\theta }]\hfill \\ & + k[\left(4b-2L_x\right)z+\left(4ab-L_{y}b+L_x{L}_y-2L_{x}a\right)\psi \hfill \\ & + \left(4b^2-L_{x}b+2L_x^2-2L_{x}b\right)\theta ]\hfill \\ & = F_{c1}\left(L_y-b\right)+F_{c2}\left(L_y-b\right)-F_{c3}b-F_{c4}b\hfill \end{array}$$

(3)

where $J_x$ and $J_y$ represent the moment of inertia of the system with respect to the $x$-axis and $y$-axis,$m$ = 100 kg, $M$ = 913 kg, $L_x$ = 1.6 m, $L_y$ = 1.2 m, $a$ = 0.8 m, $b$ = 0.6 m, $J_x$ = 76 kg*m², $J_y$ = 58 kg*m², $k$ = 600,000 N/m, and $\mathrm{c}$ = 100 Ns/m.

The equipment platform generates vibrations in the z-direction subjected to the excitation force $F_{ez}$. It is evident from Equations (1)–(3) that by adjusting the value of the control forces $F_{c1},F_{c2},F_{c3},F_{c4}$, the vibrations of the equipment platform can be reduced, achieving active vibration control. In this paper, the vibration exciter is supposed to generate random multi-frequency excitations, making this vibration system a typical high-dimensional high-frequency vibration control problem.

2.2. Active Control Implementation

In active vibration control, the control algorithm needs to calculate the control signal based on the system’s response. This section presents the implementation of the high-dimensional high-frequency vibration control problem, as shown in Figure 2. An acceleration sensor is installed near each of the four electromagnetic actuators to collect the acceleration signals at the respective positions. The four control signals are computed in real time by the controller running a trained reinforcement learning neural network, based on signals from the four acceleration sensors located near the actuators. The signal conditioner is used to filter and amplify the signals from the four acceleration sensors, which are then converted into digital signals via an AD module and fed into the controller for processing. The control signals are amplified by a driver into control current signals after being processed via a DA module to drive the actuators. The electromagnetic actuator consists of a permanent magnet armature and a coil, with the permanent magnet armature positioned in the middle of the coil. When current flows through the coil, the permanent magnet armature moves up and down under the influence of the coil’s magnetic field, generating actuating force. The driver is a power amplifier that amplifies the control signal calculated by the controller into current to drive the electromagnetic actuator to operate.

Since the vibration control system is often applied in complex vibration environments, where theoretical models cannot accurately describe the input–output response of the system, real experimental data are used for the research in this paper. In the following sections, the vibration control system is modeled as a simulator constructed from these experimental data. Based on the simulator, a neural network vibration controller is designed using reinforcement learning methods to solve the high-dimensional high-frequency vibration control problem.

3. Method

3.1. Vibration System Simulator

Figure 3 shows the vibration control simulator built using the finite-impulse response (FIR) filter. This simulator is used to describe the real vibration control system shown in Figure 1, involving five inputs and four outputs. Among them, the five inputs represent the force of the exciter and four electromagnetic actuators; the four outputs correspond to four acceleration signals. Before establishing the simulator, narrowband Gaussian white-noise force signals ranging from 0 to 100 Hz are separately inputted into the exciter and the four actuators, and experimental data from the four acceleration sensors are pre-collected. Subsequently, the input force signals and the collected acceleration data are used for the identification process of the FIR filters. These filters make up the simulator, which is capable of simulating the responses of the four acceleration sensors in the real vibration system by applying specified force signals to the exciter and electromagnetic actuator.

3.1.1. FIR Filter

Adaptive FIR filters are used to construct the simulator. As shown in Figure 4, in order to realize the FIR filter, we need to obtain the input signal $x\left(n\right)$ and the desired input and output signals $d\left(n\right)$, $x\left(n\right)$ and $d\left(n\right)$ as vectors:

$$x\left(n\right)={\left[x\left(n\right)x(n-1)\cdots x(n-N+1)\right]}^T$$

(4)

$$d\left(n\right)={\left[d\left(n\right)d(n-1)\cdots d(n-N+1)\right]}^T$$

(5)

where $N$ is the length of the filter. The input signal $x\left(n\right)$ of a given filter is known, and the output signal $y\left(n\right)$ of the filter is expressed as follows:

$$y\left(n\right)={\omega }^T\left(n\right)x\left(n\right)$$

(6)

where n indicates the serial number of the discrete-time data, and $\omega \left(n\right)$ is the vector of the filter value.

The output signal $y\left(n\right)$ is the estimation of the desired signal $d\left(n\right)$. The error $e\left(n\right)$ between the output signal $y\left(n\right)$ and the desired signal $d\left(n\right)$ is defined as follows:

$$e\left(n\right)=d\left(n\right)-y\left(n\right)$$

(7)

By using the least-mean-square (LMS) algorithm [37], we estimate the filter weight $\omega \left(n\right)$ to minimize the error $e\left(n\right)$:

$$\omega (n+1)=\omega \left(n\right)-2\mu e\left(n\right)x\left(n\right)$$

(8)

where $\mu$ is the step size.

If the appropriate input signal $x\left(n\right)$ and the desired output signal $d\left(n\right)$ are given, it is easy to estimate the filter weight $\omega \left(n\right)$. In this paper, both the input signal and the expected output signal are experimental data.

3.1.2. Using FIR Filters to the Build Simulator

The simulator can be viewed as a five-input/four-output filter. The input signals are denoted as $x_0\left(n\right)$, $x_1\left(n\right)$, $x_2\left(n\right){, x}_3\left(n\right)$, and $x_4\left(n\right)$, where $x_0\left(n\right)$ represents the force signal of the exciter, and $x_1\left(n\right)$, $x_2\left(n\right)$, $x_3\left(n\right)$, and $x_4\left(n\right)$ represent the force signals of the four electromagnetic actuators. The required output signals are recorded as $d_1\left(n\right)$, $d_2\left(n\right)$, $d_3\left(n\right)$, and $d_4\left(n\right)$, which represent the acceleration signals, respectively. $y_j\left(n\right)$ is the estimated value of the expected output ($d_j\left(n\right)$, j = 1, 2, 3, 4).

Obviously, there are 5 × 4 = 20 channels between the input and output, each of which can be represented by an FIR filter. We can suppose that $C_{ij}$ represents a filter between $x_i\left(n\right)$ and $d_j\left(n\right)$, where i = 0, 1, 2, 3, 4 and j = 1, 2, 3, 4.

Note that $d_1\left(n\right)$ is the open-loop response under $x_0\left(n\right)$, $x_1\left(n\right)$, $x_2\left(n\right){, x}_3\left(n\right)$, and $x_4\left(n\right)$ excitation. Therefore, the estimation of $d_1\left(n\right)$ can be expressed as follows:

$$\begin{array}{cc}\hfill y_1& = {\omega }_{01}^T\left(n\right)x_0\left(n\right)+{\omega }_{11}^T\left(n\right)x_1\left(n\right)+{\omega }_{21}^T\left(n\right)x_2\left(n\right)+{\omega }_{31}^T\left(n\right)x_3\left(n\right)\hfill \\ & + {\omega }_{41}^T\left(n\right)x_4\left(n\right)\hfill \end{array}$$

(9)

where $y_1\left(n\right)$ is the estimated value of $d_1$, and ${\omega }_{i1}\left(n\right)$ is the filter weight of the filter $C_{i1}$. It can be found that $y_1\left(n\right)$ is the superposition of the outputs of five FIR filters. Similarly, the estimated values of $d_2\left(n\right)$, $d_3\left(n\right)$, and $d_4\left(n\right)$ are expressed as follows:

$$\begin{array}{cc}\hfill y_2& = {\omega }_{02}^T\left(n\right)x_0\left(n\right)+{\omega }_{12}^T\left(n\right)x_1\left(n\right)+{\omega }_{22}^T\left(n\right)x_2\left(n\right)+{\omega }_{32}^T\left(n\right)x_3\left(n\right)\hfill \\ & + {\omega }_{42}^T\left(n\right)x_4\left(n\right)\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill y_3& = {\omega }_{03}^T\left(n\right)x_0\left(n\right)+{\omega }_{13}^T\left(n\right)x_1\left(n\right)+{\omega }_{23}^T\left(n\right)x_2\left(n\right)+{\omega }_{33}^T\left(n\right)x_3\left(n\right)\hfill \\ & + {\omega }_{43}^T\left(n\right)x_4\left(n\right)\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill y_4& = {\omega }_{04}^T\left(n\right)x_0\left(n\right)+{\omega }_{14}^T\left(n\right)x_1\left(n\right)+{\omega }_{24}^T\left(n\right)x_2\left(n\right)+{\omega }_{34}^T\left(n\right)x_3\left(n\right)\hfill \\ & + {\omega }_{44}^T\left(n\right)x_4\left(n\right)\hfill \end{array}$$

(12)

Formulas (10)–(12) can be written as a reduced version of Formula (13), which is the simulator of the vibration system.

$$Y\left(n\right)=W\left(n\right)X\left(n\right)$$

(13)

$$Y(n)={\left[y_1\left(n\right)y_2\left(n\right)y_3\left(n\right)y_4\left(n\right)\right]}^T$$

(14)

$$X(n)={\left[x_0^T\left(n\right)x_1^T\left(n\right)x_2^T\left(n\right)x_3^T\left(n\right)x_4^T\left(n\right)\right]}^T$$

(15)

$$W(n)=\left[\begin{array}{ccc}{\omega }_{01}^T\left(n\right)& \cdots & {\omega }_{41}^T\left(n\right)\\ ⋮& \ddots & ⋮\\ {\omega }_{04}^T\left(n\right)& \cdots & {\omega }_{44}^T\left(n\right)\end{array}\right]$$

(16)

The simulator consists of 20 single-input/single-output filters. The matrix $W\left(n\right)$ contains the weights of 20 filters, where $W_{ij}\left(n\right)$ is the weight vector of the filter $C_{ij}$: i = 0, 1, 2, 3, 4 and j = 1, 2, 3, 4.

3.2. Algorithm Framework

We consider a standard reinforcement learning setting consisting of an agent interacting with a discrete-time step-size environment $E$. At each time step $t$, the agent can receive the observed value $s_t$, take an action $a_t$, and receive a scalar reward $r_t$. The behavior of the agent is defined according to the strategy $\pi$, which maps the state to the probability distribution of the action $\pi$:$S\to P\left(A\right)$. The environment $E$ may also be random. We model it as a state-space $S$, action-space $A=R^N$, initial-state distribution $P\left(s_t\right)$, transition dynamics $P\left(s_{t+1}\right|s_t,a_t)$ reward function $r(s_t,a_t)$. The reward is defined as the sum of future reward discounts:${ R}_t={\sum }_{i=t}^r{\gamma }^{i-t}r(s_i,a_i)$. The discount factor is $\gamma$ ϵ [0,1]. The payoff depends on the chosen behavior, and therefore depends on the strategy $\pi$, and may be random. The goal of reinforcement learning is to learn a strategy that maximizes the expected revenue of the initial distribution: $J=E_{r_i,s_iϵE,a_iϵ\pi }\left[R_1\right]$. When $t$ = i, we use the minimum value of the output of the two critic networks, i.e., the output of $Q_{1t}(s_t,a_t)$ and $Q_{2t}(s_t,a_t)$ critic networks.

$$Q_t(s_t,a_t)=min\left(Q_{1t}\right(s_t,a_t),Q_{2t}(s_t,a_t\left)\right)$$

(17)

Many of the methods in reinforcement learning utilize the recursive relationship of the Bellman equation:

$$Q^{\pi }(s_t,a_t)=E_{r_i,s_{i+1}ϵE}\left[r\right(s_t,a_t)+\gamma ·E_{a_{i+1}ϵE}[Q^{\pi }(s_{t+1},a_{t+1})\left]\right]$$

(18)

If the target policy is deterministic, we can describe it as a function $\mu :S\leftarrow A$, thus avoiding internal expectations.

$$Q^{\pi }(s_t,a_t)=E_{r_i,s_{i+1}ϵE}\left[r\right(s_t,a_t)+\gamma ·Q^{\mu }(s_{t+1},\mu \left(s_{t+1}\right)\left)\right]$$

(19)

Expectations depend only on the environment. This means that the transfer generated by different random behavior strategies $\beta$ can be used to learn the $Q^{\mu }$ offline strategy. We consider a function approximator with ${\theta }^Q$ as a parameter and optimize it by minimizing the loss:

$$L({\theta }^Q)={E_{s_tϵE,a_tϵ\pi ,r_tϵE}[Q^{\mu }(s_{t+1},a_t|{\theta }^Q)-y_t]}^2$$

(20)

$$y_t=r_t+\gamma ·min(Q_{1_{t+1}}(s_{t+1},a_{t+1}),Q_{2_{t+1}}(s_{t+1},a_{t+1}))$$

(21)

Here, $r_t$ is the environmental reward, $\gamma$ is the discount factor, and $t+1$ is the index of the next state.

As shown in Figure 5, the algorithm used in this paper is mainly divided into two parts: the yellow part represents the environment (simulator) and experience playback, and the virtual part is the Actor–Critic (AC) network. Specifically, first, the environment interacts with the AC network and stores multiple data ${(s}_t,a_t,s_{t+1})$ into the experience pool. Here, $s_t$ denotes the state at time $t$, $a_t$ denotes the action at time $t$, and $s_{t+1}$ denotes the state at time $t+1$. Each data point in the experience pool is given an initial priority. When the data reach a certain amount, priority data sampling $B$ is performed for the data in the experience pool. $s_t$ is passed to the Actor network, and the Actor network is passed to the Critic a network and the Critic b network together with $s_t$, based on $s_t$‘s $a_t$. At the same time, $s_{t+1}$ is transmitted to the Actor Target network, and the $a_{t+1}$ obtained by the Actor Target network based on $s_{t+1}$ is transmitted to the Critic Target c network and the Critic Target d network together with $s_{t+1}$. The task of the Critic networks is to score the actions obtained from the Actor network. Here, we use four Critic networks, where the Critic a network and the Critic b network take the minimum value, and the Critic Target c network and the Critic Target d network also take the minimum value. Finally, the error is calculated using the Formulas (20) and (21). The dotted arrow represents the feedback neural network, and the Actor network and Critic network are updated in this way. The Actor Target network and Critic Target network adjust the neural network parameters through weight transfer. The final error is calculated using the priority formula and then used to update the priority of the data in the experience pool. When the amount of experience pool data is large enough, older data can be removed to free up space.

In terms of updating the parameters, the critic network parameter update is as follows:

$${\mathsf{\nabla }}_{{\theta }_1}J=\frac{1}{N}{\sum }_{i=1}^M(y_i-Q_{1i}(s_i,a_i\left)\right)·{\mathsf{\nabla }}_{{\theta }_1}{Q}_{1i}(s_i,a_i)$$

(22)

$${\mathsf{\nabla }}_{{\theta }_2}J=\frac{1}{N}{\sum }_{i=1}^N(y_i-Q_{2i}(s_i,a_i\left)\right)·{\mathsf{\nabla }}_{{\theta }_2}{Q}_{2i}(s_i,a_i)$$

(23)

The Actor network updates the strategy according to the feedback of the critic network, that is, the gradient of the Q-value function:

$${\mathsf{\nabla }}_{{\theta }_{\mu }}J=-\frac{1}{N}{\sum }_{i=1}^N{Q}_{1i}(s_i,{\mu }_{{\theta }_{\mu }}(s_i\left)\right)·{\mathsf{\nabla }}_{{\theta }_{\mu }}{\mu }_{{\theta }_{\mu }}\left(s_i\right)$$

(24)

where ${\mu }_{{\theta }_{\mu }}(s_i)$ represents the output action of the Actor network to the state $s_i$.

In addition, the Actor Target network and Target network update is as follows:

$${\theta }_i^{\prime }\leftarrow \tau {\theta }_i+(1-\tau ){\theta }_i^{\prime }$$

(25)

$${\mu }^{\prime }\leftarrow \tau \mu +(1-\tau ){\mu }^{\prime }$$

(26)

where i is the index of the Critic Target network and $\tau$ is the hyperparameter representing the weight of the update.

3.3. Actor–Critic (AC) Network

In this section, we focus on the AC network, where the Actor and Critic structures are the same, so we only need to explain one network structure. In order to make the network lighter and have a stronger generalization ability, we initially considered the application of convolution. However, in general, convolution requires multi-layer stacking, and too many neural network layers can easily lead to performance degradation, which may lead to gradient disappearance or gradient explosion. In order to solve this challenge, we considered and adopted the ResNet method [27]. Since we are dealing with vibration control problems, we further considered the variant deep residual shrinkage network (DRSN) [30] using ResNets.

3.3.1. Classic ResNet Architecture

The ResNet approach an emerging deep learning method that has attracted much attention in recent years [27]. Remaining building units (RBUs) are the basic components. As shown in Figure 6a, an RBU consists of two BNs, two ReLUs, two convolutional layers, and a unit shortcut. The identity shortcut is the part that makes ResNets superior to convnets in general. In general, in convolutional neural networks, the cross-entropy error gradient is propagated backward layer by layer. Using the identity shortcut, the gradient can effectively flow to the earlier layer near the input layer, so that the parameters can be updated more effectively. Figure 6b,c shows the RBUs that produce output eigenmaps of different sizes. The aim of reducing the width of the output feature map is to reduce the amount of calculation for the next layer, and the aim of increasing the number of channels of the output feature map is to facilitate the integration of different features into discriminant features. Figure 6d shows the overall architecture of a ResNet, which consists of an input layer, a convolution layer, multiple RBUs, a BN, a ReLU, a GAP, and an output full-connection (FC) layer.

3.3.2. The Basic Architecture Design of DRSNs

In this section, the motivation for developing DRSNs is introduced, and the architecture of two developed DRSNs is described in detail, including a deep residual shrinkage network with channel-shared thresholds (DRSN-CS) and a deep residual shrinkage network with channel-wise thresholds (DRSN-CW).

(1)

Theoretical background

In the past 20 years, soft threshold segmentation has often been used as a key step in many signal denoising methods [37,38]. The advantage of this method is that it transforms the original signal into a domain where the number of near-zeros is not important, and then applies the soft threshold to convert the near-zero feature to zero. In order to ensure the good performance of signal denoising, a key task of the wavelet threshold method is to design a filter to transform useful information into very positive or negative features, and transform noise information into near-zero features. However, designing such filters requires a lot of signal processing expertise and has always been a challenging problem. Deep learning provides a new way to solve this problem. Deep learning is not designed by experts, but uses the gradient descent algorithm to automatically learn the filter. The formula for the soft threshold can be expressed as follows:

$$y=\left\{\begin{array}{cc}x+\xi & x>\xi \\ 0& -\xi \le x\le \xi \\ x+\xi & x<-\xi \end{array}\right.$$

(27)

Within this formula is the input feature, the output feature, and the threshold, that is, the positive parameter. The soft threshold method does not set the negative features in the ReLU activation function to zero, but sets the features close to zero to zero, thereby retaining useful negative features.

It can be seen that the derivative of the output to the input is either 1 or 0, which can effectively prevent gradient disappearance and explosion problems. The derivative can be expressed as follows:

$$\frac{{\partial }_y}{{\partial }_x}=\left\{\begin{array}{cc}1& x>\xi \\ 0& -\xi \le x\le \xi \\ 1& x<-\xi \end{array}\right.$$

(28)

In a classical signal denoising algorithm, it is usually difficult to set an appropriate value for the threshold. In addition, the optimal value varies from case to case. To solve this problem, the threshold used in the developed DRSNs is automatically determined in the deep architecture, avoiding the trouble of manual operation. In the following chapters, the method of determining the threshold in the developed DRSNs will be introduced.

(2)

Developed DRSN-CS framework

The developed DRSN-CS is a variant of the ResNet, which uses soft thresholds to remove noise-related features. The soft threshold is inserted into the building unit as a nonlinear transformation layer. In addition, the value of the threshold can be learned in the building unit, as described below.

As shown in Figure 7a, unlike the RBU in Figure 6a, RSBU-CS has a dedicated module for soft threshold estimation. In this special module, Global Average Pooling (GAP) is performed on the absolute values of the feature map to obtain a one-dimensional vector. Then, the one-dimensional vector is propagated into a two-layer FC network to obtain a scaling parameter, similar to [39].

The sigmoid function is applied at the end of the two-layer FC network so that the scaling parameters are scaled to the range of (0, 1), which can be expressed as follows:

$$\alpha =\frac{1}{1+e^{-z}}$$

(29)

From this equation, we can find the output of the two-layer FC network in the RSBU-CS, and the delay parameter is the corresponding scaling parameter. After that, the scaling parameter delay is multiplied by the average value of the $\left|x\right|$ score to obtain the threshold. The reason for this arrangement is that the threshold of the soft threshold should not only be positive, but also not too large. If the threshold is greater than the maximum absolute value of the feature map, the output of the soft threshold is zero. In summary, the thresholds used in the RSBU-CS are expressed as follows:

$$\xi =\alpha ·\underset{i,j,c}{\mathit{average}}\left|x_{i,j,c}\right|$$

(30)

Here, $\alpha$ is the threshold, and i, j, and c are the width, height, and channel index of the feature map, respectively. The threshold can be kept within a reasonable range so that the output of the soft threshold will not be all zeroes. The two-step RSBU-CS with a doubled number of channels can be constructed similar to the RBUs in Figure 6b,c.

The brief architecture of the developed DRSN-CS is shown in Figure 7b, which is similar to the classic ResNet in Figure 6d. The only difference is that the RSBU-CS is used as a building block instead of RBUs. Multiple RSBU-CSs are stacked in the DRSN-CS to gradually reduce noise-related features. Another advantage of the DRSN-CS is that the thresholds are automatically learned in the deep architecture, rather than being manually set by experts. Therefore, professional signal processing knowledge is not required to implement the developed DRSN-CS.

(3)

Developed DRSN-CW framework

The developed DRSN-CW is another variant of a ResNet. Unlike the DRSN-CS, it applies a separate threshold to each channel of the feature map, which will be introduced below. Figure 7c shows the RSBU-CW. Through absolute operation and the GAP layer, the upper and lower lines of the feature map $x$ are reduced to one-dimensional vectors, and they are propagated to the two-layer FC network.

$${\alpha }_c=\frac{1}{1+e^{-z_c}}$$

(31)

Here, z_c is the feature at the cth neuron, and ${\alpha }_c$ is the cth scaling parameter. After that, the threshold can be calculated:

$${\xi }_c={\alpha }_c·\underset{i,j}{\mathit{average}}\left|x_{i,j,c}\right|$$

(32)

where i, j, and $c$ denote the width, height, and channel index of the feature map x, respectively. Similar to the DRSN-CS, the threshold can be positive and maintained within a reasonable range to prevent the output feature from being all zeros.

The overall architecture of the developed DRSN-CW is shown in Figure 7d. By superimposing multiple RSBU-CWs, through various nonlinear transformations, the soft threshold is used as the shrinkage function to learn the discriminant features and eliminate the noise-related information.

(4)

Detailed Network Architecture of the Actor

In Figure 8, there are dashed- and solid-line skips. When the input and output dimensions are the same (solid-line skips), residual learning is directly applied, i.e., $x+f\left(x\right)$, where $f\left(x\right)$ represents the output result of the construction unit. In cases where the dimensions increase (dashed-line skips), matching dimensions occur (achieved by 1 × 1 convolution) with a stride of 2. Finally, through Avgpooling and flattening, we obtain [B, 32], which, via FC, leads to [B, 4] dimensions, representing the action signals from four sensors. The structure of the Critic’s four networks resembles that of the Actor network, differing only in the input and output. The input is $\left\{s_t,a_t\right\}[$ = {[B, 1, 4, 100], [B, 4]}, and the output is [B, 1]. Here, the input data are preprocessed as [B, 1, 4, 101], signifying 100 dimensions for each sensor’s data and the action’s value.

3.4. Multi-Reward Mechanism

The multi-reward mechanism is a technology in reinforcement learning that is used to introduce and process multiple reward signals more effectively to enhance the learning and decision-making ability of agents. In some complex environments, a single reward signal may not fully express the diversity and complexity of the environment. Therefore, the introduction of multiple reward signals helps to provide more comprehensive and accurate training signals to improve the learning process and strategies. Compared with the previous method [26], the reward function is as follows:

$$r_{OM}=-(x_1^2+x_2^2+x_3^2+x_4^2)$$

(33)

We propose three other reward functions, as follows (34)–(36):

$$r_{MSV}=-\frac{1}{N}·\sum _{i=1}^N(x_{1i}^2+x_{2i}^2+x_{3i}^2+x_{4i}^2)$$

(34)

As shown in Formula (34), considering the mean square of four sensors over a period of time as the reward signal helps to reduce noise and fluctuations, making it smoother. This aids the agent to better understand the long-term trends of the environment, avoiding excessive interference from individual time-step rewards, and improving the stability and reliability of training. This approach also helps the agent to better consider long-term gains, reinforcing awareness of long-term planning and decision making rather than just focusing on immediate rewards. Averaging the reward signal also reduces the impact of environmental noise and randomness, enhancing the agent’s ability to handle complex situations. Overall, adopting the average value over a period of time as the reward function can improve training stability, smooth the reward signal, better consider long-term gains, and enhance the agent’s resistance to interference, effectively improving the training effectiveness of reinforcement learning models.

$$\begin{array}{c}{r}_{RMSE}=-\frac{1}{M}·{\displaystyle \sum _{j=1}^M}\sqrt{\frac{{\sum }_{i=1}^N{x}_{ji}^2}{N}}\\ =-\frac{1}{M}·\left(\sqrt{\frac{{\sum }_{i=1}^N{x}_{1i}^2}{N}}+\sqrt{\frac{{\sum }_{i=1}^N{x}_{2i}^2}{N}}+\sqrt{\frac{{\sum }_{i=1}^N{x}_{3i}^2}{N}}+\sqrt{\frac{{\sum }_{i=1}^N{x}_{4i}^2}{N}}\right)\end{array}$$

(35)

As shown in Formula (35), adopting the average of the root mean square (RMSE) of four sensors as the reward function has several advantages. Firstly, compared to extreme values, it exhibits higher robustness and stability. This means that the RMSE has less impact on the reward function when dealing with outliers, more evenly considers the overall characteristics of the data, and is less affected by extreme values. Additionally, the RMSE, being the square root of the mean square error, provides a more intuitive and understandable measurement. Compared to the mean square error, RMSE offers a more intuitive description of the error magnitude, emphasizing the size of the average error rather than the square of errors. It is noteworthy that the RMSE is less influenced by noise because it takes the square root of the mean error, resulting in a smoother signal. In certain situations, the RMSE better reflects real-world error scenarios, aligning more closely with the background and requirements of actual tasks. Lastly, due to its wide applicability, the RMSE as a reward function is suitable for various problems and scenarios and is commonly used in performance evaluation. In conclusion, selecting the RMSE as the reward function can provide a more robust, smooth, and understandable reward signal for training, helping the agent learn and adapt to the environment more effectively.

$$r_v=-\sum _{j=1}^M\frac{{\sum }_{i=1}^N{(x_{ji}-{\overline{x}}_j)}^2}{N}$$

(36)

As shown in Formula (36), adopting the average of the variance of the four sensors as the reward function has unique advantages. It is a statistical measure of data dispersion, providing a metric for the diversity of behaviors exhibited by the agent in reinforcement learning. By evaluating the degree of behavioral variation, variance helps us to understand the consistency and diversity of the agent’s actions across different states. Additionally, variance also reflects the stability of the agent’s policy execution. A smaller variance may indicate more consistent and predictable behavioral patterns, while a larger variance implies more diversified actions taken across different states. Variance also indicates the agent’s sensitivity to environmental changes, and its adaptability to environmental changes can also be measured through variance. Overall, variance as a reward function provides a quantitative measure of the diversity, stability, and adaptability of the agent’s behavior in reinforcement learning. By assessing behavioral changes, variance helps us to gain insight into the agent’s learning process and strategy adjustments, guiding its optimization in complex environments.

We combine the influence of four different reward functions on the control strategy. Through comparative experiments, the optimal reward method is obtained, which can be seen in the subsequent analyses.

3.5. Priority Experience Replaying

The motivation of introducing priority experience replaying is to optimize the training process and solve the problems in traditional experience replaying. The traditional replaying method treats all empirical samples equally, but in fact, some experience is more important for model learning. Priority experience replaying introduces a mechanism that can focus more frequently on learning those experiences that have a greater impact on the model, thereby improving the efficiency of training and the performance of the model. In order to overcome these problems, we introduce a random sampling method [40] between pure greedy priority and uniform random sampling. We ensure that the probability of being sampled in the priority of a transition is monotonic, and that the probability is non-zero even for the transition of the lowest priority. Specifically, we define the probability i of the sampling transition as follows:

$$P(i)=\frac{P_i^{\alpha }}{{\sum }_k{p}_k^{\alpha }}$$

(37)

where $P_i$ > 0 is the priority of conversion i. The index $\alpha$ determines how much priority is used, and $\alpha$ = 0 corresponds to the uniform case.

The first variant we consider is a direct, proportional priority, where $P_i=\left|{\delta }_i\right|+\epsilon$, and $\epsilon$ is a small positive constant that prevents the transition edge from not being revisited once the error is zero. The second variant is an indirect, rank-based priority, where $P_i=\frac{1}{rank\left(i\right)}$, and $rank\left(i\right)$ is the level of data i sorted by $\left|{\delta }_i\right|$ in the experience pool. In this case, $P$ becomes a power-law distribution with exponent $\alpha$. In error, both distributions are monotonic, but the latter is more robust because it is not sensitive to outliers.

In order to effectively sample from the distribution in Equation (37), the complexity cannot depend on the capacity $N$ of the empirical pool. For rank-based variants, we can approximate the cumulative density function using a piecewise linear function with k equal probability segments. The segment boundaries can be calculated in advance (they only change when $N$ or $\alpha$ changes). At runtime, we sample a fragment and then uniformly sample it in the transformation. This is especially effective when combined with minibatch-based learning algorithms: selecting $k$ as the size of the minibatch and sampling a transition from each segment precisely is a form of stratified sampling that has the additional advantage of balancing the minibatch. The proportional variables are different, and also allow effective implementation based on the “sum-tree” data structure (where each node is the sum of its child nodes, and the priority is the leaf node), which can be effectively updated and sampled.

4. Experimental Results and Discussion

This chapter mainly introduces some comparative experiments and ablation experiments, including lightweight comparative experiments, priority experience replaying comparative experiments, optimal reward function combination, demonstrating the feasibility of this lightweight method, alleviating overestimation experiments, and relevant comparative experiments.

4.1. Lightweight Comparative Experiment

Under narrowband white-noise excitation ranging from 0 to 100 Hz, with 0.3 s of data for root mean square calculation,

Table 1. Lightweight comparison experiment. Under the same experimental conditions, the method proposed in this paper achieves better control effectiveness with fewer parameters. The first row represents the algorithm, the second row displays the vibration signal values before being controlled, the third row shows the vibration signal values after being controlled, the fourth row indicates the parameter quantity, and the fifth row represents the reduction in vibration control, measured in dB.

Algorithm	Uncontrolled (V)	Controlled (V)	Parameter (N)	Reduction (dB)
FC	0.3311	0.0762	507398	12.750
CNN	0.3311	0.0870	3102254	11.604
GoogleNet	0.3311	0.0812	15774938	11.894
LSTM	0.3311	0.0875	375326	11.516
DRSL-MPER	0.3311	0.0328	67554	20.240

illustrates the setup of platform simulators for vibration control using our DRSL-MPER algorithm and under the DDPG framework using fully connected (FC), CNN, GoogleNet, and LSTM methods. The first column indicates various algorithms; the second column shows the values of the vibration signal before being controlled, measured in volts (V); the third column displays the values of the vibration signal after being controlled, also measured in volts (V); the fourth column indicates the parameter quantity, measured as a number (N), representing the total number of weight parameters and bias parameters in the neural network; and the fifth column represents the degree of vibration signal reduction, measured in decibels (dB).

Reference [26] proposed the fully connected network and DDPG framework as the primary points of comparison, detailing their parameters and vibration reduction levels. Considering the prevalent use of fully connected network methods in the current domain, we introduced methods such as convolutional neural networks (CNNs), GoogLeNet, and Long Short-Term Memory networks (LSTMs) to expand the experimental scope; relevant parameters and vibration reduction levels are listed in the table above. Under narrowband white-noise excitation ranging from 0 to 100 Hz, our DRSL-MPER method achieved a vibration reduction of up to 20.240 dB with a reduction in parameters by more than 7.5 times. A greater reduction indicates better vibration control effectiveness.

4.2. Comparison between Prioritized Experience Replaying and Non-Prioritized Experience Replaying

To clearly present the experimental results, we visualized the time-domain signals before and after the control, the frequency-domain signals obtained through Fourier transformation, the action-value signals, and the reward values. Similar experimental results are also summarized here for subsequent experiments. As shown in Figure 9, the first row (a)–(d) displays the time-domain control signals, where the x-axis represents time and the y-axis represents the control signal in volts (V); the second row (e)–(f) illustrates the frequency-domain control signals, with the x-axis representing the frequency range in Hz and the y-axis indicating the values after Fourier transformation (FFT); the third row (i)–(l) represents the action values of the agent; and the fourth row (m) represents the reward values. In the latter two rows, the x-axis represents time and the y-axis represents the magnitude of the action values or the number of rounds for the reward values. Although reward values may be negative, higher values indicate better control effectiveness, albeit not absolutely. These visualizations help demonstrate the performance of the algorithm during the control process and the variations in signals.

Figure 10 illustrates how the utilization of prioritized experience replay enhances the convergence speed and control effectiveness under both DRSN and TD3 reinforcement learning frameworks. Under narrowband white-noise excitation ranging from 0 to 100 Hz, Figure 10a shows that optimal performance is reached after approximately 2500 iterations, while Figure 10b shows that the same is achieved after only 1300 iterations, showcasing a convergence speed close to twice that of non-experience replay. To validate the effectiveness of prioritized experience replay, we conducted training with and without prioritized experience replay for 1000 episodes, as shown in Figure 10c,d, respectively. Through calculations, Figure 10c demonstrates a vibration reduction of 7.9 dB without prioritized experience replay, while Figure 10d indicates a reduction of up to 12.5 dB with prioritized experience replay. These contrasting experiments reveal that training with prioritized experience replay is more stable and exhibits higher data efficiency.

4.3. Optimal Reward Function Combination

To explore the most suitable combination of reward functions, we synthesized the original reward function, variance, root mean square, and mean from [26], as detailed in

Table 2. In the cases of TD3 and DRSN, with combinations of multiple reward functions, we denote the variance, mean square value, root mean square, and original method as V, MSV, RMS, and OM, respectively. In the same experimental setting, the OM reward function exhibits negative effects, while the root mean square generates positive impacts.

V, MSV, RMS, OM	Reduction (dB)
V	15.988
MSV	16.032
RMS	16.284
OM	13.722
V, MSV	3.496
V, RMS	16.094
V, OM	12.062
MSV, RMS	20.240
MSV, OM	3.518
RMS, OM	9.238
V, MSV, RMS	16.402
V, MSV, OM	15.598
V, RMS, OM	2.234
MSV, RMS, OM	15.074
V, MSV, RMS, OM	13.858

. Under the conditions of employing both DRSN methods in the Actor and Critic networks, along with the application of prioritized experience replay, we experimented with various combinations of reward functions and drew conclusions. When used individually, the original reward function performed the poorest compared to variance, root mean square, and mean. Among the combined functions, as highlighted in red, the combination with the root mean square exhibited the most favorable effects. However, the data highlighted in blue indicated that the original reward function might have a detrimental impact on the combination, often leading to decreased performance. Therefore, we recommend using the combination of root mean square and mean values, which performs better in most cases, resulting in the maximum reduction of up to 20.240 dB.

4.4. Demonstrating the Feasibility of This Lightweight Method

To verify the feasibility of the lightweight method, we compared the performance of the fully connected networks under different settings. In this experiment, the Actor network structure was [400, x, x, 4], and the Critic network structure was [400, x, x, 1], where x is an integer. For both Figure 11a,b, we gradually increased the size of the neural network’s intermediate layers from 8 to 16, to 32, to 64, to 128, doubling each layer. In Figure 11a, the different colors represent different numbers of intermediate layers in the Critic network. As the number of intermediate layers in the Actor network increased, the effectiveness of vibration control gradually improved. In Figure 11b, the different colors represent different numbers of intermediate layers in the Actor network. As the number of intermediate layers in the Critic network increased, the effectiveness of vibration control also increased. In our DRSL-MPER method, the Actor and Critic networks used the same network structure, with parameter counts of 22,584 and 44,970, respectively, achieving a control effect of 20.24 dB. In contrast, when the intermediate layer size was 64, the Actor had 30,084 parameters, the Critic had 59,906 parameters, and the control effect reached 5.358 dB, which is close to our method in terms of the parameter count. This indicates that DRSL-MPER increased the damping effect by 14.882 dB with fewer parameters. In comparison, the best control effect occurred when the intermediate layers of both the Actor and Critic were 256, achieving a control effect of 15.57 dB. The Actor had 169,476 parameters, the Critic had 337,922 parameters, and DRSL-MPER improved the control effect by 4.67 dB with only 13% of the parameters of the 256-layer network. These results clearly demonstrate the significant advantages of our lightweight network in relation to both its effectiveness and its parameter count.

4.5. Alleviating Overestimation Experiments

This study adopts the TD3 algorithm, aiming to alleviate the common problem of value function overestimation in the Actor–Critic architecture. The Actor–Critic algorithm combines value function approximation (Critic) and policy approximation (Actor), where the Critic is responsible for estimating state-value or action-value functions, and the Actor updates the policy based on these estimates. However, due to errors in function approximation, nonlinear approximations, and limited sample sizes, the Critic network in the Actor–Critic algorithm often overestimates the true value, leading to suboptimal or even unstable policies. This experiment compared the DDPG framework with the TD3 framework. We recorded their Q-values and true values across 10,000 episodes. In Figure 12, the horizontal axis represents the episodes, and the vertical axis represents the magnitude of Q-values. Figure 12a shows the Q-values and true values of DDPG. It is observed that the Q-values are consistently higher than the true values, which also proves the existence of overestimation in the AC framework. Next, we introduced the TD3 framework, and in Figure 12b, it is shown that the Q-values of TD3 overlap significantly with the true values, indicating that the TD3 framework effectively alleviates the overestimation problem.

4.6. Relevant Comparative Experiments

4.6.1. Relevant Comparative Experiments for Lightweightness

In Section 4.1, we emphasized the feasibility and advantages of the lightweighting approach. To gain further insights into the impact of lightweighting experiments on different modules and parameter quantities, we conducted comparative experiments specifically focusing on the Actor, Critic, and reinforcement learning frameworks. The baseline model comprised Actor and Critic networks with fully connected layers of a 256 size (FC₂₅₆), utilizing the DDPG reinforcement learning framework. In

Table 3. The impact of Actor and Critic networks as well as the reinforcement learning framework in the lightweighting comparative experiments. By controlling the Actor network, Critic network, RL framework, and parameter count, it was observed that in both the DDPG and TD3 frameworks, the influence of the Critic network on control effectiveness outweighs that of the Actor network.

Actor	Critic	RL	Parameter (N)	Reduction (dB)
FC256	FC256	DDPG	338437	12.722
DRSN	FC256	DDPG	191545	13.102
FC256	DRSN	DDPG	191961	14.727
DRSN	DRSN	DDPG	45069	12.966
FC256	FC256	TD3	507398	14.276
DRSN	FC256	TD3	360506	16.196
FC256	DRSN	TD3	214446	13.014
DRSN	DRSN	TD3	67554	20.240

, the first column represents the Actor network structure, the second column denotes the Critic network structure, the third column indicates the reinforcement learning framework, the fourth column details the total parameter count, and the fifth column presents the reduction, where smaller values are preferred. The experimental findings reveal that the DRSN performs better than FC₂₅₆ in both the Actor and Critic networks. Additionally, despite the TD3 algorithm increasing the parameter count under the same network structure, there is a significant enhancement in the control effectiveness, and the increased parameter count remains within an acceptable range. Furthermore, we observed that improvements in the Critic network have a more pronounced impact on control effectiveness compared to improvements in the Actor network. The best vibration reduction can reach up to 20.240 dB.

4.6.2. Relevant Comparative Experiments for Prioritized Experience Replay

In Section 4.2, we emphasized the superiority of prioritized experience replay under the DRSN and TD3 reinforcement learning frameworks. To further demonstrate its generalizability, we conducted multiple experiments. Figure 13a–f demonstrate the enhanced data utilization efficiency by prioritized experience replay under both fully connected and DRSN conditions. Both (a) and (b) adopt the DDPG framework under the DRSN. In (a), the number of rounds taken to reach convergence under prioritized experience replay is around 350, while in (b), convergence is achieved at approximately 700 rounds. Both (c) and (d) utilize the TD3 framework under the fully connected network. In (c), the number of rounds taken to achieve convergence under prioritized experience replay is approximately 350, while in (f), convergence is not reached even after 1750 rounds. Both (e) and (f) adopt the DDPG framework under the fully connected network. In (e), the number of rounds taken to reach convergence under prioritized experience replay is approximately 400, with a vibration reduction of 9.25 dB, while in (f), convergence is achieved at approximately 600 rounds, with a vibration reduction of 3.53 dB. Both the DDPG and TD3 reinforcement learning algorithms exhibit effectiveness, highlighting that prioritized experience replay is not constrained by specific neural network architectures or reinforcement learning frameworks. It fundamentally enhances data utilization efficiency, proving its generalizability. While Figure 13a–f differ in the network structures, Figure 13 showcases more pronounced control effects, indirectly validating the feasibility of the DRSN method.

4.6.3. Relevant Comparative Experiments for the Optimal Reward Functions

In Section 4.3, we discussed the optimal reward function combinations within the DRSN and TD3 frameworks. To further investigate the applicability of these reward functions, we conducted additional studies under baseline experimental conditions, namely within the FC₂₅₆ and DDPG frameworks. In Table 2 and

Table 4. In the cases of DDPG and FC₂₅₆, with combinations of multiple reward functions, we denote the variance, mean square value, root mean square, and original method as V, MSV, RMS, and OM, respectively. We assign a rating to the reward functions, utilizing measurements before and after the control process.

V, MSV, RMS, OM	Reduction (dB)
V	10.458
MSV	11.990
RMS	13.506
OM	9.156
V, MSV	1.938
V, RMS	13.386
V, OM	6.504
MSV, RMS	13.524
MSV, OM	2.256
RMS, OM	5.43
V, MSV, RMS	13.096
V, MSV, OM	11.508
V, RMS, OM	1.284
MSV, RMS, OM	10.526
V, MSV, RMS, OM	9.318

, similar trends can be observed. The data points highlighted in red indicate the superior effectiveness of combining the root mean square function. Conversely, the data points highlighted in blue suggest that the original reward function may adversely impact the combined performance, often resulting in inferior outcomes. Therefore, we recommend employing a combination of the root mean square and mean square value functions.

4.6.4. Ablation Experiments

In this section, ablation experiments were conducted on a narrowband white-noise excitation dataset ranging from 0 to 100 Hz to verify the effectiveness of the proposed modules.

Table 5. In the ablation experiments, OM, MSV, RMS, and PER, respectively, represent the original method, mean square value, root mean square, and prioritized experience replay. The ablation experiments demonstrated the efficiency of DRSN lightweighting and the MSV + RMS combination function, with a faster convergence speed for PER.

Algorithm	Average Convergence Episodes	Parameter	Reduction (dB)
Baseline DDPG (FC + ROM)	2031	338437	12.728
DDPG (DRSN + ROM)	1562	45069	15.328
DDPG (DRSN + RMSV+RMS)	1468	45069	16.210
DDPG (DRSN + RMSV+RMS + PER)	634	45069	18.108
DRSL-MPER (TD3 + DRSN + RMSV+RMS + PER)	652	67554	20.240

illustrates the various methods studied alongside average convergence episodes, parameters, and reduction levels. We noticed that compared to the baseline, the introduction of the DRSN not only reduced the parameters by approximately 6.5 times but also increased the damping effect by 2.6 dB. Furthermore, the inclusion of the R_MSV+RMS reward function resulted in an additional damping improvement of 0.882 dB. Subsequently, the introduction of prioritized experience replay reduced the average convergence rounds to 634 times. Lastly, the integration of the TD3 framework, although it increased the parameters and rounds slightly, did not affect the overall performance, yet achieved the peak control effectiveness of 20.240 dB.

5. Conclusions

This work introduces a multi-reward mechanism to assist lightweight networks in finding nearly optimal control strategies. Additionally, a prioritized experience replay mechanism is introduced to assign priority to data, accelerating the convergence speed of neural networks and enhancing data utilization efficiency. The incorporation of a deep residual contraction network achieves adaptive denoising and lightweighting of the neural networks, making it currently one of the most preferable methods under real-world conditions. In these experiments, we utilized a continuous length of 100 time points from each sensor as the state, and our plans for future research involve further exploration of appropriate time point lengths for the sensors. Furthermore, the consideration of convolutional hyperparameters is also deemed worthwhile. Future research directions should encompass migrating this method to force control and magnetic control, as well as registration and classification tasks in medical imaging. Through these efforts, we aim to continuously optimize and advance the application of this method in various domains.