A Deep Reinforcement Learning Approach to Injection Speed Control in Injection Molding Machines with Servomotor-Driven Constant Pump Hydraulic System

Abstract

The control of the injection speed in hydraulic injection molding machines is critical to product quality and production efficiency. This paper analyzes servomotor-driven constant pump hydraulic systems in injection molding machines to achieve optimal tracking control of the injection speed. We propose an efficient reinforcement learning (RL)-based approach to achieve fast tracking control of the injection speed within predefined time constraints. First, we construct a precise Markov decision process model that defines the state space, action space, and reward function. Then, we establish a tracking strategy using the Deep Deterministic Policy Gradient RL method, which allows the controller to learn optimal policies by interacting with the environment. Careful attention is also paid to the network architecture and the definition of states/actions to ensure the effectiveness of the proposed method. Extensive numerical results validate the proposed approach and demonstrate accurate and efficient tracking of the injection velocity. The controller’s ability to learn and adapt in real time provides a significant advantage over the traditional Proportion Integration Differentiation controller. The proposed method provides a practical solution to the challenge of maintaining accurate control of the injection speed in the manufacturing process.

1. Introduction

Injection molding is a process that involves injecting molten plastic or metal through the injection device of an injection molding machine (IMM) into a mold. After cooling and solidification, the process results in the formation of the final product. The basic principles encompass four stages: melting, injection, cooling, and demolding [1,2,3]. In an IMM, as illustrated in Figure 1, plastic raw material undergoes heating and melting processes, transforming into a molten state. Subsequently, through a precise injection mechanism, the molten material is injected into the mold cavity, forming the specific shape of the desired part or product. Throughout this process, controlling key parameters such as the injection speed, temperature, pressure, and cooling time is crucial to ensure that the dimensional accuracy, surface quality, and mechanical performance of the final product meet the design requirements.

Regulating the injection speed stands as a critical element in the injection molding process, directly influencing the quality and productivity of molded items. The primary objective of injection speed control is to uphold a consistent flow rate of molten material throughout the injection phase, thereby avoiding the occurrence of defects, such as flow marks, shrinkage, and stress. During high-speed filling, molten material experiences substantial frictional heating as it flows into the cavity via the nozzle and runner system, causing a rise in its temperature. However, an insufficient injection speed prolongs the filling time. Conversely, filling the mold cavity at a high injection speed may lead to unsatisfactory welds at the rear of the insert, consequently diminishing the strength of the product. Therefore, achieving optimal tracking control of the IMM speed is a crucial means of improving production efficiency and reducing production costs in the injection molding process. Controlling the injection speed presents a challenge due to the dynamic nature of the molten material’s physical properties, including the viscosity and specific volume, as well as quality parameters, like the temperature and pressure. These properties fluctuate over time, location, and process conditions, contributing to the nonlinearity and uncertainty inherent in molten material flow.

In the developmental trajectory of the IMM control field, scholars both domestically and internationally have conducted extensive research on speed optimization and control methods for various injection molding systems over the past few decades [4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]. Currently, in the practical field of the injection molding industry, the widely applied and mature methods are still predominantly focused on traditional control methods. For example, in [21], Tan et al. proposed a learning-enhanced PI control method for the periodic control of the ram speed in IMMs. In [12], Yang et al. introduced an enhanced PID method to improve injection speed control. Yang et al. introduced an improved PID method utilizing unsaturated integration and differential elimination for enhanced injection speed control. In [22], Tian et al. employed a dual-controller scheme for injection speed control, allowing for setpoint tracking and load suppression, demonstrating excellent system robustness. Special techniques such as profile transformation and shifting were also introduced to improve speed response. In [7], Huang et al. proposed neural network-based predictive learning control for injection molding ram speed control. Wu et al. [19] tackled the speed servo system in an IMM by employing a first-order Taylor series to enhance and approximate the dynamic model. Subsequently, leveraging the linear–quadratic regulator (LQR) method, they designed a robust feedback controller. In recent years, several comparatively intelligent control methods have been suggested. For instance, Xu et al. [17] presented a method combining deep neural networks with traditional optimal control methods to achieve velocity control of the leading edge position of the molten material flow inside the IMM, thereby improving the efficiency of plastic production. Ren et al. [18] explored the issue of speed tracking control for a hydraulic cylinder within a standard servomotor-driven IMM. They proposed an effective two-stage optimization framework, leveraging the combination of control parameterization and particle swarm optimization to generate optimal signal inputs, achieving the desired injection speed tracking within a given time frame. Although these control methods have demonstrated good dynamic performance and made certain achievements in improving system stability, when faced with IMM systems possessing highly nonlinear and complex system dynamics, traditional speed control methods such as conventional PID control and empirical model control may exhibit suboptimal performance. This is especially true when control parameters need to be predetermined, leading to degradation in control performance. Additionally, high-performance injection speed control requires strong real-time capability and robustness, which may pose a challenge for certain control methods. Therefore, the current trend is moving toward more advanced control methods, particularly those based on deep learning and RL, to enhance the performance of IMM injection speed control.

In recent years, deep reinforcement learning (DRL) has emerged as a thriving control method [23,24,25,26,27]. The Deep Deterministic Policy Gradient (DDPG) algorithm [23], as a representative of DRL, combines the powerful approximation capabilities of deep learning with the advantages of policy gradient methods. This capability enables it to more effectively adapt to highly nonlinear and intricate systems, presenting a fresh approach to tackling the control hurdles faced by IMMs characterized by high-dimensional and continuous action spaces. The uniqueness of the DDPG algorithm lies in its adaptability to continuous action spaces, its ability to learn the dynamics of nonlinear systems, and the practicality of real-time policy adjustments. In comparison to traditional methods, the DDPG can learn the dynamic characteristics of a system from data without the need for an explicit system model. This flexibility provides a versatile solution for dealing with nonlinearity and uncertainty in real-world production scenarios, opening up new possibilities for more complex injection molding industrial processes. Currently, the DDPG has demonstrated successful applications in various domains, including robot control [28,29,30], autonomous driving [31,32,33,34,35], and industrial processes [36,37,38,39,40]. For tasks like optimal tracking control of IMM speed that require highly precise control and real-time responsiveness, the introduction of the DDPG brings new possibilities for improving production efficiency and reducing energy consumption.

This paper focuses on a nonlinear hydraulic servo control system within a standard IMM, aiming to examine the optimal tracking control of the injection speed throughout the filling phase of the molding process. We propose and design an effective optimal controller based on the DDPG method, achieving rapid tracking control of the injection speed within a given time frame. In practical applications, ensuring consistency between the actual injection speed and the desired trajectory poses a challenge due to the multifaceted influences of various factors on speed dynamics. Thus, attaining precise and robust control of the injection speed is pivotal for enhancing product quality. In this study, our primary aim is to devise an effective learning control approach for swiftly and reliably tracking the injection speed in a standard IMM, leveraging advanced DRL techniques. To address this problem, specifically, we first establish a Markov decision process (MDP) model oriented toward injection molding speed control within the framework of DRL algorithms. This includes the concrete definition and design of the state space, action space, and reward function. By using key parameters of the injection molding process as state inputs, the controller is able to perceive and adapt to continuously changing production conditions. Injection speed adjustment is defined as the output action of the controller. Based on this foundation, we design an injection speed tracking control strategy using the DDPG algorithm, enabling the controller to learn the optimal control policy through interaction with the environment. In the controller design process, we carefully select the network structure and define states and actions to ensure that the DDPG algorithm can learn effective control strategies in the complex injection molding environment. Finally, the viability and efficacy of the proposed approach are comprehensively verified through extensive experimental simulations. The results demonstrate that the proposed algorithm adeptly achieves rapid and effective tracking of the desired injection speed, surpassing the performance of the traditional PID control method commonly employed in injection machines. To our knowledge, this study represents the first application of RL algorithms to speed control within the injection molding process. The proposed algorithm framework also holds significant promise for various applications in the field.

In summary, this paper presents innovative contributions in the following three aspects:

• A nonlinear hydraulic servo control system in a typical IMM is formulated and the optimal tracking control of the injection speed throughout the filling phase of the molding process is studied.
• An efficient MDP model is established for injection molding speed control, encompassing the definition and design of the state space, action space, and the specific formulation of the reward function. Subsequently, an efficient optimal controller based on the DDPG is devised to achieve swift and precise tracking control of the injection speed within a predefined time frame.
• Extensive numerical experiments are performed to comprehensively confirm the feasible and efficient properties of the proposed method. Furthermore, a comparative analysis with the traditional PID algorithm is carried out, offering additional evidence to underscore the superiority of the proposed algorithm.

The structure of this paper is outlined as follows: Section 2 presents a dynamic mathematical model for a typical hydraulic servo speed system in injection molding equipment, facilitating subsequent training of RL algorithms. The optimal tracking control problem of the injection speed in the hydraulic servo system is established within this section. In Section 3, an MDP model tailored for injection molding speed control is developed, incorporating specific definitions and designs for the state space, action space, and reward function. Section 4 introduces the design of the corresponding control algorithm based on the DDPG. Section 5 further validates the proposed controller through simulations, with an additional investigation into the robustness of the proposed controller in the presence of disturbances. Finally, the paper concludes with a summary and prospects for future work.

2. Problem Formulation

2.1. Dynamic Model of the Hydraulic Servo Speed System

In the hydraulic servo speed system of the injection molding equipment, the variation in the stamping displacement is primarily determined by the stamping velocity:

$$\dot{p}\left(t\right)=v\left(t\right),$$

(1)

where $p\left(t\right)$ represents the stamping displacement, and $v\left(t\right)$ represents the stamping velocity.

During the actual filling process, the stamping velocity is related to the hydraulic injection pressure, nozzle pressure, and stamping displacement. The relationship among them can be expressed as follows:

$$\begin{array}{cc}\hfill \dot{v}\left(t\right)& ={\displaystyle \frac{1}{M}}\left[(P_1\left(t\right)A_1-P_2\left(t\right)A_2)-(2\pi \eta R^{1-n}(L+p\left(t\right)){\left({\displaystyle \frac{(s-1)v}{K_r^{1-s}-1}}\right)}^n)\right],\hfill \end{array}$$

(2)

where $P_1\left(t\right)$ represents the hydraulic injection pressure, $P_2\left(t\right)$ represents the nozzle pressure, $A_1$ is the cross-sectional area of the cylinder, $A_2$ is the cross-sectional area of the barrel, $K_r$ is the ratio of the machine radius to the nozzle radius, L is the initial length of the screw, n is the power-law index of the polymer melt, R is the nozzle radius, M is the mass of the actuator-screw assembly, s is the reciprocal of the power-law index of the polymer melt, and $\eta$ is the viscosity of the polymer.

During the filling process, the hydraulic injection pressure is related to the hydraulic oil flow rate entering the hydraulic cylinder, stamping displacement, and stamping velocity, as expressed by the following:

$${\dot{P}}_1\left(t\right)={\displaystyle \frac{{\beta }_1}{{\alpha }_{10}+A_{1}p\left(t\right)}}\left(u\left(t\right)-A_{1}v\left(t\right)\right),$$

(3)

where ${\beta }_1$ represents the hydraulic fluid bulk modulus, ${\alpha }_{10}$ represents the oil volume on the injection side, $u\left(t\right)$ is the control input representing the hydraulic oil flow rate entering the injection cylinder, and $A_1$ represents the cross-sectional area of the cylinder.

Additionally, the nozzle pressure is related to the stamping injection velocity and the polymer flow rate. The nozzle pressure can be determined by the following equation:

$${\dot{P}}_2\left(t\right)={\displaystyle \frac{{\beta }_2}{{\alpha }_{20}-A_{2}p\left(t\right)}}\left(A_{2}v\left(t\right)-Q_p\right),$$

(4)

where ${\beta }_2$ represents the nozzle volume modulus, ${\alpha }_{20}$ represents the volume of polymer in the barrel, $A_2$ represents the cross-sectional area of the barrel, and $Q_p$ represents the polymer flow rate.

2.2. Initialization of Initial Conditions in Process Control Systems

At the start of the filling process, we define the initial (or startup) state of the dynamic system as follows:

$$\begin{array}{c}\hfill p\left(0\right)=p_0,v\left(0\right)=v_0,P_1\left(0\right)=P_{10},P_2\left(0\right)=P_{20},\end{array}$$

(5)

In the above expressions, $p_0,v_0,P_{10}$, and $P_{20}$ are predetermined values, and their numerical ranges are determined by the initial state values of the actual IMM.

2.3. Control Objective

The quality of injection-molded products is heavily influenced by the IMM’s capacity to precisely track the desired injection speed within a designated time frame T. The central aim of this paper is to devise an optimal control signal $u\left(t\right)$ to swiftly track the desired injection speed while minimizing energy consumption. To this end, we introduce the following performance cost function:

$$\underset{u\left(t\right)}{min}:J\left(t\right)={\displaystyle \frac{1}{2}}{\int }_0^T\left[{(v\left(t\right)-v_d\left(t\right))}^2+u^2\left(t\right)\right]dt.$$

(6)

Here, $v_d\left(t\right)$ represents the desired injection speed during the filling process, serving as a time-varying trajectory variable that can be predetermined based on the operational requirements of the system.

3. Construction of the Markov Decision Process (MDP) Model

In DRL, the intelligent agent engages in real-time interaction with the environment, aiming to select optimal actions based on reward feedback in the current state. Typically, this task is referred to as an MDP. The MDP comprises the fundamental elements $(\mathit{S},\mathit{A},\mathit{P},\mathit{R},\gamma )$, where $\mathit{S}$ represents all the possible states the system can be in, $\mathit{A}$ denotes all the possible actions the agent can take, $\mathit{P}\left(s_{t+1}\right|s_t,a_t)\to [0,1]$ signifies the probability of transitioning from the current state to the next state given a specific action, ${\mathit{R}}_t=\mathit{R}(s_t,a_t,s_{t+1})$ indicates the immediate reward at each state–action pair, and $\gamma \in [0,1]$ represents the discount factor for future rewards. At each time step, the agent observes a state $s_t$ and takes an action $a_t$, the environment responds to this action, and the agent receives a reward $r_t$, transitioning to the next state $s_{t+1}$. The overall objective of the MDP is to find a policy $\pi :\mathit{S}\to \mathit{A}$ that maximizes the cumulative reward when decisions are made based on this policy.

In the context of the injection speed tracking control problem of IMMs studied in this paper, the term “agent” refers to the designed control algorithm responsible for decision-making and learning. The “environment” encompasses external factors, such as the IMM and IMM operation commands. The term “Action” represents the adjustment strategy for the optimal input control signal $u\left(t\right)$. In this subsection, we adopt an RL framework, describing the trajectory tracking problem of the injection speed as a typical MDP. This involves constructing the state space, action space, reward function, etc., within the system. Building upon this foundation, we design a DRL algorithm to effectively control the injection speed of the IMM.

3.1. Definition of the State Space

Within the framework of the Markovian property, the construction of the state space aims to encompass all the information about the environment. The current state is determined solely by the preceding state and executed actions. In the context of our IMM study, the machine’s state primarily comprises essential parameters such as the current speed, temperature, and pressure, which are pivotal for comprehending the system’s dynamic behavior.

For the injection speed tracking control problem, the observable system output states are represented as $p\left(t\right),v\left(t\right),P_1\left(t\right),P_2\left(t\right)$, where $p\left(t\right)$ denotes the position, $v\left(t\right)$ denotes the velocity, and $P_1\left(t\right)$ and $P_2\left(t\right)$ are two key pressure parameters. Therefore, we define the state space $\mathit{S}$ as follows:

$$\begin{array}{cc}\hfill \mathit{S}& =〈p\left(t\right),v\left(t\right),P_1\left(t\right),P_2\left(t\right)〉.\hfill \end{array}$$

(7)

3.2. Definition of the Action Space

The action space ${\mathit{A}}_t$ is defined as a set of various actions that influence the injection speed, including acceleration, deceleration, and maintaining the current speed. The selection of actions plays a crucial role in the performance and stability of the system. Specifically, the action space ${\mathit{A}}_t$ can be represented as the set of control inputs $u\left(t\right)$, which is multidimensional and varies according to practical needs.

Thus, the action space ${\mathit{A}}_t$ is defined as follows:

$$\begin{array}{c}\hfill {\mathit{A}}_t=〈u\left(t\right)〉.\end{array}$$

(8)

This definition of the action space allows us to adjust the injection speed by manipulating the control input $u\left(t\right)$. This strategy enables the direct mapping of observed results to the platen speed command, and using a direct command permits the IMM to have a high-speed response and an extremely rapid correction speed.

3.3. Definition of the Reward Function

The reward function reflects the immediate rewards obtained by the agent at each state–action pair. In the context of this work, our optimization objective is based on a cost function that incorporates the velocity error and input energy, as shown in Equation (6). To enhance the agent’s sensitivity and agility in tracking error, we first define the error reward function $R_{e\left(j\right)}$ as follows:

$$\begin{array}{ccc}\hfill R_{e\left(j\right)}& =& {∥{\mathit{v}}_{\mathit{d}}\left(\mathit{t}\right)-\mathit{v}\left(\mathit{t}\right)∥}_2^2,\hfill \end{array}$$

(9)

where ${\mathit{v}}_d\left(t\right)$ represents the desired velocity, and $\mathit{v}\left(t\right)$ represents the velocity at the current position.

Simultaneously, to further ensure energy efficiency, we define the input reward function $R_{u\left(j\right)}$ as follows:

$$\begin{array}{ccc}\hfill R_{u\left(j\right)}& =& -\mathit{u}{\left(j\right)}^T\Omega \mathit{u}\left(j\right),\hfill \end{array}$$

(10)

where $\Omega$ is a positive definite weight matrix.

Therefore, at each time step j, we define the overall reward function as follows:

$$\begin{array}{ccc}\hfill R_f\left(j\right)& =& -{\rho }_1{R}_{e\left(j\right)}-{\rho }_2{R}_{u\left(j\right)},\hfill \end{array}$$

(11)

where ${\rho }_1$ and ${\rho }_2$ are predefined weight coefficients.

4. Optimal Velocity Tracking Strategy Based on the DDPG Algorithm

4.1. Deep Deterministic Policy Gradient (DDPG) Algorithm

In the field of RL, the classical Q-learning algorithm [41] has demonstrated remarkable performance in handling discrete action spaces. However, as the demand for continuous action spaces has increased, Q-learning has exhibited certain limitations. In response to this challenge, the DDPG algorithm has emerged as a robust solution for addressing continuous action spaces. The DDPG is an RL algorithm that combines deep learning with deterministic policy gradient methods, primarily employed to tackle issues related to continuous action spaces. The DDPG algorithm achieves a comprehensive and effective solution for RL problems in continuous high-dimensional action spaces through the incorporation of experience replay, the collaborative training of actor and critic networks, and the introduction of target networks. In recent years, the DDPG has found widespread applications across various domains [28,29,30,31,32,34,36,37,38,39,40], yielding significant results.

The objective of the DDPG algorithm is to maximize the sum of the discounted future rewards J, defined as follows:

$$\begin{array}{c}\hfill J_t=r_t+\gamma r_{t+1}+{\gamma }^2{r}_{t+2}+\cdots =\sum _{k=0}^{\infty }{\gamma }^k{r}_{t+k}\end{array}$$

(12)

where $\gamma \in [0,1]$ represents the discount factor.

The core principles of the DDPG algorithm include the actor network and the critic network, as illustrated in Figure 2. Fundamentally, this approach is a hybrid that merges policy gradient and value function techniques. The policy function, termed the actor, and the value function Q, referred to as the critic, are integral components. The actor network generates continuous actions, while the critic network assesses the value function of state–action pairs. The actor network optimizes the policy by maximizing the expected action-value function, while the critic network evaluates the value of actions by minimizing the mean squared error between estimated values and actual returns. This combination demonstrates excellent performance in handling continuous action spaces. Both networks are represented by deep neural networks, described by parameter sets ${\theta }^{\mu }$ and ${\theta }^Q$, respectively. The objective of the actor network is to learn a deterministic policy $\mu \left(s\right|{\theta }^{\mu })$, where s is the state and ${\theta }^{\mu }$ is the parameter of the actor network. The actor network updates its parameters by maximizing the expected return:

$${\theta }^{\mu }\leftarrow arg\underset{{\theta }^{\mu }}{max}J\left({\theta }^{\mu }\right)={\mathbb{E}}_{s_t\sim {\rho }^{\beta }}\left[Q(s_t,\mu \left(s_t\right|{\theta }^{\mu }))\right]$$

Here, $J\left({\theta }^{\mu }\right)$ represents the objective function of the actor network, indicating the expected action value. $s_t$ represents the current state, $\rho$ denotes the state distribution, and $\mu \left(s\right|{\theta }^{\mu })$ represents the deterministic policy generated by the actor network. $Q(s_t,\mu \left(s_t\right|{\theta }^{\mu }))$ is the value function of the state–action pair evaluated by the critic network.

The critic network is employed in RL to assess the value function of a given state–action pair, denoted as $Q(s,a|{\theta }^Q)$. Throughout the training process, the objective of the critic network is to minimize the mean squared error of the Bellman equation. The aim of the critic network is to assess the value function for state–action pairs, denoted as $Q(s,a|{\theta }^Q)$, where ${\theta }^Q$ represents the parameters of the critic network. The critic network adjusts its parameters by minimizing the following Bellman equation:

$$\begin{array}{cc}\hfill {\theta }^Q& \leftarrow arg\underset{{\theta }^Q}{min}J\left({\theta }^Q\right)=\mathbb{E}{s}_t,a_t,r_{t+1},s_{t+1}\sim D\left[(Q(s_t,a_t|{\theta }^Q)\right.\hfill \\ \hfill & -\left.{(r_{t+1}+\gamma Q(s_{t+1},\mu \left(s_{t+1}\right|{\theta }^{\mu })|{\theta }^{Q^{\prime }}))}^2\right]\hfill \end{array}$$

(13)

Here, $J\left({\theta }^Q\right)$ represents the objective function of the critic network, indicating the mean squared error between the estimated value and the actual return. D denotes the experience replay buffer, $r_{t+1}$ signifies the immediate reward, and $Q(s_t,a_t|{\theta }^Q)$ denotes the output of the critic network, representing the estimated action value. Additionally, $\mu \left(s_{t+1}\right|{\theta }^{\mu })$ represents the deterministic policy generated by the target network for the next state, and ${\theta }^{Q^{\prime }}$ denotes the parameters of the target network.

In the DDPG, the value function for state–action pairs, denoted as $Q(s,a|{\theta }^Q)$, is represented by the critic network. This is defined according to the Bellman equation as follows:

$$\begin{array}{c}\hfill Q(s,a|{\theta }^Q)={\mathbb{E}}_{(r,s^{\prime })\sim \mathcal{E}}[r+\gamma Q(s^{\prime },\mu \left(s^{\prime }\right|{\theta }^{{\mu }^{\prime }})|{\theta }^{Q^{\prime }})].\end{array}$$

(14)

Moreover, $\mathcal{E}$ represents the experience replay buffer. The DDPG introduces an experience replay mechanism to decouple correlations in continuous trajectories, thereby enhancing the algorithm’s stability and sample efficiency.

To improve stability, the DDPG also incorporates target networks, including the target policy network ${\mu }^{\prime }$ and the target value function network $Q^{\prime }$. The parameters of these target networks are periodically updated through a soft update mechanism, defined as follows:

$$\begin{array}{ccc}\hfill {\theta }^{Q^{\prime }}& \leftarrow & \tau {\theta }^Q+(1-\tau ){\theta }^{Q^{\prime }}\hfill \\ \hfill {\theta }^{{\mu }^{\prime }}& \leftarrow & \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }}.\hfill \end{array}$$

(15)

Here, $\tau$ is a small constant, where $\tau \ll 1$, used to control the magnitude of the updates. This soft update strategy contributes to the gradual adjustment of the target network parameters, promoting algorithmic stability.

4.2. Training Procedure of DDPG

The training process of the DDPG primarily consists of two steps: a policy update and value function update. The DDPG updates its policy network by maximizing the action-value function $Q(s,\mu \left(s\right|{\theta }^{\mu })|{\theta }^Q)$. The actor network undergoes updates by applying the chain rule to the expected return of the initial distribution J. The optimization objective is structured as follows:

$$\begin{array}{cc}\hfill {\nabla }_{{\theta }^{\mu }}J\approx & {\mathbb{E}}_{s\sim {\rho }_s}[{\nabla }_{a}Q(s,a|{\theta }^Q){|}_{s=s_t,a=\mu \left(s_t\right)}{\nabla }_{{\theta }^{\mu }}\mu \left(s\right|{\theta }^{\mu }){|}_{s_t}].\hfill \end{array}$$

(16)

The DDPG updates the value function network by minimizing the mean squared error between the value function network and the target value function network. The optimization objective is expressed as follows:

$$\begin{array}{ccc}\hfill Loss\left({\theta }^Q\right)& =& \mathbb{E}\left[{\left(y_j-Q\left(s_j,a_j\mid {\theta }^Q\right)\right)}^2\right].\hfill \end{array}$$

(17)

Following the definition of the action value, the current action value $y_j$ can be represented as follows:

$$\begin{array}{ccc}\hfill y_j& =& r_j+\gamma Q^{\prime }\left(s_{j+1},{\mu }^{\prime }\left(s_{j+1}\mid {\theta }^{{\mu }^{\prime }}\right)\mid {\theta }^{Q^{\prime }}\right).\hfill \end{array}$$

(18)

It is important to highlight that in both the estimated and actual reward elements, the current action value $y_j$ proves to be more precise compared to the evaluation derived from the Q network. The modification of network parameters can be depicted as follows:

$$\begin{array}{ccc}\hfill \delta {\theta }^Q& =& \tau ·\nabla Loss\left({\theta }^Q\right)\hfill \end{array}$$

(19)

In the DDPG, the $\mu$ network is divided into two parts: the current time-step $\mu$ network ($\mu \left(s_t\right)$) and the previous time-step $\mu$ network (${\mu }^{\prime }\left(s_t\right)$). The current $\mu$ network is used for action execution and network updates, while the previous $\mu$ network is used to compute the action $a_{t+1}$ for the next time step. The update of the $\mu$ network depends on the action value, with the objective of maximizing the action value. Therefore, the corresponding update equation is as follows:

$$\begin{array}{}\mathrm{(20)}& \hfill \delta {\theta }^U& =& \tau ·\nabla Q\left(s_t,a_t\mid {\theta }^Q\right)\hfill \mathrm{(21)}& & =& {\left.\tau ·{\nabla }_{\theta }U\left(s_t\right)·{\nabla }_{a}Q\left(s_t,a\mid {\theta }^Q\right)\right|}_{a=U\left(s_t\right)}\hfill \end{array}$$

Furthermore, Gaussian noise is incorporated into the network output to regulate the exploration rate of the network. Thus, the network output ${\widehat{a}}_t$ can be described as follows:

$$\begin{array}{c}\hfill {\widehat{a}}_t\sim \mathcal{N}\left(a_t,{\sigma }^2\right)\end{array}$$

(22)

where ${\sigma }^2$ represents the variance of the Gaussian noise.

4.3. Velocity Tracking Control Strategy Based on DDPG

In this section, a velocity tracking optimization control strategy is proposed based on the DDPG algorithm. The overall algorithmic structure is illustrated in Figure 3. In this framework, the network $\mu$ is tasked with receiving the current system state s and generating the action a. The action controller, after passing through a Gaussian distribution and action constraints, exerts control over the injection speed of the IMM. The network Q is responsible for representing the action value, evaluating the quality of action selection. Figure 4 provides a detailed depiction of the structures of the $\mu$ and Q networks.

The neural network framework adopts an end-to-end policy, mapping the observation space to the action space, akin to mapping desired output values to given parameters for the controller input. In the diagram, the forward propagation process of the actor neural network can be observed, transforming input states through a series of linear transformations and nonlinear activation functions to produce the output action. Specifically, in the experiments conducted in this paper, the $\mu$ network has two hidden layers, each with 256 nodes. The activation function for the hidden layers is ReLU, and the output layer is linear. In the architecture of the Q network, two hidden layers comprising 256 nodes each are established. ReLU serves as the activation function for these hidden layers, while the output layer employs a Tanh activation function. The DDPG involves both the actor and the critic neural networks collaborating. The actor network generates actions, while the critic evaluates the value associated with these actions. The training of this deep learning model typically involves using gradient descent methods to minimize a certain loss function, optimizing network parameters to generate better action policies and Q value estimates. The training process of the velocity tracking control strategy based on the DDPG is outlined in Algorithm 1.

Algorithm 1 Training process of injection speed tracking control strategy via DDPG.

1: Initialize the IMM state and read the desired velocity file.   2: Initialize the replay buffer $\mathcal{D}$.   3: Initialize actor network parameters ${\theta }^{\mu }$ and critic network parameters ${\theta }^Q$.   4: Initialize target actor network ${\theta }^{{\mu }^{\prime }}\leftarrow {\theta }^{\mu }$ and target critic network ${\theta }^{Q^{\prime }}\leftarrow {\theta }^Q$.   5: Initialize exploration noise $\mathcal{N}$.   6: for episode $=1$ to M do   7:    Reset the environment state s.   8:    for each time step T do   9:       Choose action $a_t=\mu \left(s\right|{\theta }^{\mu })+{\mathcal{N}}_t$ with exploration noise. 10:      Execute action $a_t$, observe reward $r_t$, and new state $s^{\prime }$. 11:      Store transition $(s,a_t,r_t,s^{\prime })$ in the replay buffer $\mathcal{D}$. 12:      Sample random minibatch of transitions $(s_j,a_j,r_j,s_j^{\prime })$ from $\mathcal{D}$. 13:      Compute target value: $y_j=r_j+\gamma Q(s_j^{\prime },\mu \left(s_j^{\prime }\right|{\theta }^{{\mu }^{\prime }})|{\theta }^{Q^{\prime }})$. 14:      Update critic network by: $L={\displaystyle \frac{1}{N}}{\sum }_j{(Q(s_j,a_j|{\theta }^Q)-y_j)}^2$. 15:      Update actor network using: ${\nabla }_{{\theta }^{\mu }}J\approx {\displaystyle \frac{1}{N}}{\sum }_j{\nabla }_{a}Q(s,a|{\theta }^Q){{|}_{s=s_j,a=\mu \left(s_j\right)}{\nabla }_{{\theta }^{\mu }}\mu \left(s\right|{\theta }^{\mu })|}_{s_j}$ 16:      Update target networks: ${\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }},{\theta }^{Q^{\prime }}\leftarrow \tau {\theta }^Q+(1-\tau ){\theta }^{Q^{\prime }}$ 17:      Update ${\sigma }^2$. 18:    end for 19: end for

5. Experimental Results

This section presents a series of simulation experiments conducted to confirm the feasibility and effectiveness of the proposed algorithm. The primary focus was on assessing the tracking control performance of the injection speed in the IMM using the DDPG optimal feedback controller developed. Initially, the RL model was trained based on the provided target state trajectory to acquire and produce optimal control policies. This enabled the achievement of tracking control for the desired injection speed trajectory within the specified time frame. Secondly, in the experimental section, we extensively discussed and analyzed the impact of hyperparameter tuning on the experimental results during the algorithm training process. Finally, we compared the experimental results of the proposed control strategy algorithm with the traditional PID control algorithm, further confirming the superiority of our proposed method. Throughout the entire experimental process, all the simulations were conducted in the Matlab R2023b simulation environment.

Table 1. Main parameter settings.

Variable	Variable Name	Value
Weight of Actuator-Screw Assembly	M	8.663 kg
Cross-sectional Area of Cylinder	$A1$	3342 mm2
Cross-sectional Area of Barrel	$A2$	201 mm2
Polymer Viscosity	$\eta$	4600 Pa·s
Nozzle Radius	R	2 m
Initial Length of Screw	L	0.1 m
Ratio of Unit Radius to Nozzle Radius	$K_r$	0.9
Bulk Modulus of Hydraulic Fluid	${\beta }_1$	1020 MPa
Nozzle Bulk Modulus	${\beta }_2$	1100 MPa
Oil Volume on Jet Side	${\alpha }_{10}$	17,045 mm3
Polymer Volume in Barrel	${\alpha }_{20}$	11,678 mm3
Polymer Flow Velocity	$Q_p$	0.0011/min

presents the pertinent characteristic parameters of the IMM filling model examined in this paper.

5.1. Validation of DDPG-Based Controller

In this subsection, we conducted tracking control experiments for different injection speed targets $v_d\left(t\right)$ in the IMM. The primary objective was to thoroughly validate the feasibility and effectiveness of the controller based on the DDPG algorithm proposed in this paper.

Initially, we focused on tracking control simulation experiments for the first set of desired IMM injection speed targets $v_d\left(t\right)$. We defined a two-stage injection speed trajectory to implement the optimal tracking controller design based on the DDPG algorithm. The final numerical results are illustrated in Figure 5. From the simulation results in Figure 5, it is evident that as the DDPG algorithm training progresses, the learning and decision networks gradually learn the optimal control strategy. This ultimately leads to the rapid and accurate convergence of the injection speed output of the dynamic injection molding system to the target speed curve. During the optimal controller design process, the DDPG algorithm determines the control gains in each control policy iteration by iteratively reinforcing the action-value function in the RL framework. In Figure 6, we provide a scatter plot of the control action values. Figure 7 illustrates the convergence progression of the average reward curve throughout the training of the algorithm. From the result figures, it can be observed that in the initial training stage of the algorithm, the action values and rewards fluctuate significantly, possibly reflecting the initial uncertainty of the system to the environment and the continuous trial-and-error process of learning. However, as the training progresses, we observe that the action values and rewards gradually stabilize. After a period of learning, the system can adjust its action strategy to better adapt to the specific requirements of the IMM. This indicates that the evolution process of training is not only the result of reducing action value fluctuations but also reflects a better understanding of the task goals by the system.

To further validate the effectiveness of the proposed controller based on the DDPG algorithm, we also conducted optimal tracking control simulation experiments for the given three-stage injection speed curves. The corresponding numerical results are shown in Figure 8, Figure 9 and Figure 10. Similar to the results of the two-stage injection speed trajectory tracking, as the DDPG algorithm training progresses, the learning and decision networks gradually learn the optimal control strategy. This ultimately leads to the rapid and accurate convergence of the injection speed output of the system to the target speed curve. The simulation results also affirm the effectiveness of the controller designed through the DDPG algorithm. Furthermore, we conducted optimal tracking control simulation experiments for four-stage injection speed curves, and the corresponding numerical results are presented in Figure 11, Figure 12 and Figure 13. Similar to the results of the two-stage and three-stage injection speed trajectory tracking, the simulation results also confirm the effectiveness of the controller designed through the DDPG algorithm.

Additionally, during the experimental process, it was demonstrated that with the extension of the training time of the reinforcement learning algorithm, the system’s error in reaching the target speed gradually decreased. Through longer training times, the model or system could more effectively adapt to the environment, learn more accurate control strategies, and exhibit more stable and accurate performance when achieving the target speed.

5.2. Hyperparameter Tuning

During the experimental process, we conducted a study on the hyperparameters of the algorithm, testing and analyzing the impact of different hyperparameter settings on the experimental results. The specific analysis of the experimental results is presented below.

5.2.1. Gaussian Noise in Action Space

During training, Gaussian noise is incorporated into the output actions of the network. This assists in introducing a degree of randomness into the training procedure. This comprehensive learning of the environment model encourages the agent to explore the environment more effectively, enhancing the algorithm’s robustness and generalization. The variance ${\sigma }^2$ plays a pivotal role in defining the extent of the exploration space. Altering the values of different variances ${\sigma }^2$ results in varying convergence speeds during training. When ${\sigma }^2$ exceeds 0.3, the network fails to converge within the training period, leading to training instability. As shown in Figure 14, when ${\sigma }^2$ is within the range of 0.02–0.30, the network exhibits good stability. Larger values of ${\sigma }^2$ lead to slower convergence.

5.2.2. Discount Factor $\gamma$ in the Reward Function

The discount factor $\gamma$ in the reward function has a significant impact on DRL algorithms. When $\gamma$ approaches 1, the diminishing effect of future rewards is reduced, and the agent pays more attention to long-term rewards. This inclination encourages the agent to make decisions that lead to higher cumulative rewards in the long run. Conversely, when $\gamma$ approaches 0, the diminishing effect of future rewards is enhanced, and the agent focuses more on immediate rewards. This tendency leads the agent to make decisions that result in immediately higher rewards, without much consideration for long-term consequences. In the context of the speed tracking task, the impact of different discount factors $\gamma$ on reward convergence is investigated. As shown in Figure 15, the convergence curves of rewards under different discount factors $\gamma$ indicate that the average reward does not vary significantly as $\gamma$ increases from 0.1 to 0.999. The findings indicate that the network demonstrates a strong tolerance for a wide range of $\gamma$ values.

5.2.3. Soft Update Parameter $\tau$ for Updating Target Network

The parameter $\tau$ is responsible for controlling the proportion of new weights introduced during updates. When $\tau$ is small, the introduction of new weights to the main network is gradual, resulting in smoother updates. On the other hand, when $\tau$ is large, updates occur more rapidly. Selecting an appropriate $\tau$ value often requires experimentation and adjustment during the training process. As depicted in Figure 16, when $\tau$ increases from 0.005 to 0.5, the convergence of the reward curve becomes slower with the increasing $\tau$. However, the algorithm still effectively tracks the expected speed curve. This observation suggests that a careful choice of $\tau$ is crucial for achieving a balance between stability and convergence speed.

5.2.4. Batch Size for Training

The batch size, representing the number of samples randomly drawn from the experience replay buffer during each model parameter update, is another crucial hyperparameter in this experiment. To optimize the replay buffer, experiments were conducted using five different batch sizes within the range of [10–500]. As shown in Figure 17, the experimental results indicate that as the batch size for training increases, the convergence becomes faster, leading to improved controller performance.

5.2.5. Learning Rate

For the DDPG, a deep reinforcement learning algorithm suitable for continuous action spaces, a small learning rate is commonly chosen. As illustrated in Figure 18, the comparison of the experimental results reveals that a smaller learning rate results in smaller update magnitudes and a potentially slower convergence of rewards. However, a smaller learning rate leads to a more stable model that can better adapt to noise and variations, resulting in a more effective tracking of the expected speed curve.

5.3. Compared with PID Method

To further validate the performance of the proposed DDPG-based RL controller for the speed tracking task in the system, more experiments were conducted for comparing our proposed control algorithm with the widely used PID control method, as well as the Ziegler–Nichols and Cohen–Coon tuning methods. In the field of speed tracking control for IMMs, PID control is widely applied due to its simplicity and intuitive design. Ziegler–Nichols (ZN) and Cohen–Coon (C-C) are two common methods for PID parameter tuning, designed to optimize control performance. However, these traditional control methods demonstrate significant limitations when applied to complex, high-dimensional dynamic systems.

In the comparative experiments, the control performance of the PID (including the ZN and C-C methods) is compared with that of the proposed DDPG-based controller in tracking the injection speed profile. The results, as shown in Figure 19, reveal that in the three-stage injection speed profile tracking task, the PID-based control schemes exhibited noticeable oscillations at certain points. This issue was especially evident with the ZN and C-C methods, which struggled to prevent significant overshoot and response instability. In contrast, the DDPG-based controller demonstrated superior stability and adaptability to dynamic changes. The DDPG controller was capable of adjusting its strategy in real time, resulting in smoother speed profile tracking with minimal oscillation and overshoot. Therefore, the experimental results highlight the clear advantages of the DDPG controller in addressing the high-dimensional, nonlinear dynamic task of IMM speed tracking. Compared to the Ziegler–Nichols and Cohen–Coon methods, the DDPG not only improved control accuracy but also enhanced system robustness and stability. Particularly when handling complex nonlinear systems, the DDPG provides a more forward-looking and efficient control solution, surpassing the limitations of traditional PID methods.

6. Conclusions

This paper focuses on the nonlinear hydraulic servo control system within a standard IMM, presenting the design of an efficient optimal controller based on the DDPG algorithm. The primary objective of the controller is to achieve rapid tracking control of the injection speed within a predefined time frame. Employing the framework of deep reinforcement learning algorithms, we constructed a customized MDP model for the control of the injection molding speed. This model meticulously defines the state space, action space, and reward function. Expanding upon this model, we established an injection speed tracking control strategy based on the DDPG. This approach enables the controller to learn the optimal control policy through interaction with the environment. To validate the feasibility and effectiveness of the proposed method, we conducted extensive experimental simulations. The results demonstrate that the suggested algorithm can adeptly and efficiently track the desired injection speed. In contrast to the traditional PID method, the DDPG-based approach requires no expert tuning, resulting in smoother speed profiles and smaller speed errors. The methodology proposed in this paper exhibits promising applications and can be extended to address optimal control challenges in other intricate injection molding industrial processes. Furthermore, its applicability extends to analogous nonlinear optimal control issues in discrete manufacturing processes. Although the method proposed in this work is very effective, it should be also noted that there is still much room for improvement in the future. For example, the exploration of more offline DRL algorithms can be also pursued in future work. Moreover, more injection molding process production factors need to be taken into account. For example, in the injection molding process, the projected area and the mass of the part to be injected significantly affect the injection pressure, injection speed, cooling time, cycle time, and mold design complexity. Larger projected areas and heavier parts typically require higher injection pressure and speed, extended cooling times, increased material usage, and more complex mold designs. These factors also impact production efficiency and costs. Therefore, during design and production, these aspects should also be carefully considered to optimize the molding process and ensure the final product quality. In addition, the size of the clamping force also impacts equipment selection and energy consumption. Properly determining the clamping force is essential not only for ensuring part quality but also for optimizing production efficiency and cost-effectiveness.