Tuning of PID Controllers Using Reinforcement Learning for Nonlinear System Control

Abstract

This paper presents the application of reinforcement learning algorithms in the tuning of PID controllers for the control of some classes of continuous nonlinear systems. Tuning the parameters of the PID controllers is performed with the help of the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, which presents a series of advantages compared to other similar methods from machine learning dedicated to continuous state and action spaces. The TD3 algorithm is an off-policy actor–critic-based method and is used as it does not require a system model. Double Q-learning, delayed policy updates and target policy smoothing make TD3 robust against overestimation, increase its stability, and improve its exploration. These enhancements make TD3 one of the state-of-the-art algorithms for continuous control tasks. The presented technique is applied for the control of a biotechnological system that has strongly nonlinear dynamics. The proposed tuning method is compared to the classical tuning methods of PID controllers. The performance of the tuning method based on the TD3 algorithm is demonstrated through a simulation, illustrating the effectiveness of the proposed methodology.

1. Introduction

Control algorithms are essential to industrial automation and process control, ensuring efficient, safe, and reliable operations. These algorithms govern the behavior of machinery, systems, and processes in industries like manufacturing, biotechnology, energy, robotics, and automotives. Implementing these algorithms in hardware allows for the real-time, high-performance control of industrial systems. Hardware implementations of control algorithms in industrial applications are critical to achieving high-performance, reliable, and efficient operations. By leveraging microcontrollers, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic controllers (PLCs), and custom application-specific integrated circuits (ASICs), industries can ensure real-time responses, reduce downtime, and optimize their processes. As technology advances, the integration of AI, edge computing, and other innovations will continue to shape the future of control systems in industrial applications.

The numerical implementation of controllers allows the use of very complex algorithms with ease. However, in practice, due to its proven advantages, the proportional–integral–derivative (PID) controller (and its variants) is widely used in industrial control systems, as well as in many other applications that require continuous control. Most of the methods used to tune the parameters of PID controllers are based on time-invariant linear models of the processes, which, in practice, can lead to the poor performance of the control system. Despite the existence of a wide range of advanced control methods, most industrial processes use classical PID-type control laws as control methods. This is due to the robustness of these control laws to disturbances, to modeling errors, or to the time variations of different parameters, but also due to the simplicity of implementation on both analog and digital devices.

The following are some specific examples of the use of PID controllers.

-

In industrial processes, PID controllers are used to control the temperature, pressure, levels, and other important variables. They can help to keep these variables within safe and effective limits [1,2].

-

In the aerospace industry, PID controllers are used to control the flight of aircraft. They can help to keep the aircraft straight and on course, even in difficult conditions [3].

-

In the automotive industry, PID controllers are used to control the engine, transmission, and other systems. They can help to improve vehicles’ performance, efficiency, and safety [4].

The tuning of classic PID controllers consists of setting the values of only three parameters—Kp, Ki, and Kd—corresponding to the three actions specified in the name of these controllers—proportional (P), integral (I), and derivative (D). Although, for linear systems, the tuning of a PID controller is straightforward, using various methods [5], the practical implementation raises numerous problems because real systems are nonlinear (or the linearity zone is very narrow) or have variable parameters. Moreover, some processes allow aggressive control, while others require smooth control. Thus, the selection of the parameters must be carried out according to the specific characteristics of the process, which is why, in industrial practice, various approaches have been developed for the tuning of the parameters of PID controllers [6]. The simplest tuning method is trial and error. In this method, the parameters of the controller are adjusted according to the response of the system to various test signals. It is a method that requires experienced personnel and can lead to equipment failure if not performed carefully. In addition, in this category of methods, we can include tuning using the Ziegler–Nichols frequency response method [7]. In this case, the closed-loop system is brought to the stability limit and the parameters are adjusted based on practical rules. Another category of methods is based on obtaining a simplified first-order plus time delay model (FOPTD) and using preset tuning formulas to adjust the aggressiveness and robustness of the response to various types of input signals.

While PID controllers are particularly effective for linear systems, they can also be applied to certain nonlinear systems, with some limitations. The challenge with nonlinear systems is that their behavior may vary across different operating points, and a fixed set of PID parameters may not provide optimal control across the entire range of system dynamics [8]. In some cases, nonlinearities can lead to instability or poor performance when using a standard PID controller.

However, there are several strategies to using PID controllers with nonlinear systems, among which are linearization [8], gain scheduling [9], and adaptive control [10].

Most of these methods assume good knowledge of the system model (which must be obtained through mathematical modeling or through identification techniques), with the performance of the control system being directly influenced by the quality of this model.

Linearization: One approach is to linearize the system around an operating point and design a PID controller for this linearized model. This can work well if the nonlinearities are relatively small within the operating range.

Gain Scheduling: Another method is gain scheduling, where different sets of PID parameters are used for different operating conditions or operating points in the system’s state space. The controller parameters are adjusted based on the system’s nonlinearities.

Adaptive Control: Adaptive control techniques can be employed to adjust the PID parameters online based on the changing characteristics of the nonlinear system. This requires a feedback mechanism to continuously update the controller parameters.

This paper presents a method based on machine learning to tune the parameters of PID controllers for the automatic control of some classes of nonlinear systems. The development of machine learning methods in the field of artificial intelligence has been extended to the field of process control. Thus, the most well-known algorithms in the field of reinforcement learning (the field of AI closest to automatic control) have been tested and implemented in applications of control systems [11,12]. In particular, the actor–critic methods in RL have been intensively used for system control and optimization problems [13,14,15,16].

The first applications of RL in control systems were dedicated to discrete systems, mainly because the states and actions were discrete and finite. RL methods for the control of discrete systems, such as Q-learning or SARSA, are designed to operate in this context and may use tables to store and update the values of states and actions [17]. RL methods for the control of continuous systems often involve a number of additional challenges, such as efficiently exploring continuous action spaces, managing trade-offs between exploration and exploitation, and ensuring the convergence and stability of the algorithms in this environment. An example of an off-policy RL method used in tuning controllers is adaptive dynamic programming (ADP), which is a class of techniques used to solve optimization and decision-making problems in uncertain environments. It combines elements of reinforcement learning, dynamic programming, and function approximation [18]. ADP techniques have been successfully used in the control of nonlinear systems using various control techniques: optimal [19] and adaptive control [20] or fuzzy control [21].

Twin Delayed DDPG (TD3) is an off-policy reinforcement learning algorithm designed for continuous action spaces. It is an extension of the Deep Deterministic Policy Gradient (DDPG) algorithm, with several modifications to enhance its stability and performance. TD3 was introduced by Scott Fujimoto et al. in their 2018 paper titled, “Addressing Function Approximation Error in Actor-Critic Methods” [22], and it has been successfully used in several control applications [13,14,15,16,23].

The main novelties presented in this article are the development and design of a structure for the tuning of PID controller parameters using the TD3 algorithm, the application of this method in a strongly nonlinear system in the field of biotechnological systems, and a comparison between classical PID controller tuning methods and the reinforcement learning-based technique.

In this study, the TD3 algorithm was used to tune the parameters of the PID controller from the control loop of a nonlinear system.

This paper is structured as follows: in Section 2, the TD3 algorithm is presented in detail; Section 3 presents the tuning of the PID controller parameters using the TD3 algorithm; Section 4 presents the classical approach of PID tuning for a nonlinear system; Section 5 presents the simulation results for the two approaches; and Section 6 is dedicated to the conclusions regarding the results obtained and possible future approaches.

2. Twin Delayed Deep Deterministic Policy Gradient (TD3) Algorithm

Twin Delayed Deep Deterministic Policy Gradient (TD3) is an off-policy actor–critic reinforcement learning algorithm. TD3 is an advanced reinforcement learning (RL) algorithm specifically designed for continuous action spaces. It is a successor (an enhanced version) to the Deep Deterministic Policy Gradient (DDPG) algorithm, and it addresses some of the shortcomings of DDPG, such as overestimation bias and numerical instability [24]. Since TD3 is built on DDPG, with some modifications, we will present the DDPG algorithm first.

2.1. Basics of Reinforcement Learning Algorithms

Reinforcement learning (RL) is a type of machine learning where an agent learns to behave in an environment by trial and error. The agent receives rewards for taking actions that lead to desired outcomes and punishments for taking actions that lead to undesired outcomes. Over time, the agent learns to take actions that maximize its rewards. RL is a powerful technique that can be used to solve a wide variety of problems, including game playing, robotics, or finance [11].

RL is a challenging field of machine learning, but it is also one of the most promising. RL has the potential to solve problems that are beyond the reach of other machine learning techniques. Some of the key concepts in reinforcement learning are the following (see also Figure 1) [24].

• Agent: The agent is the entity that is learning to behave in an environment. The most common structure for the agent is composed of two elements: the critic and actor. The critic estimates the expected cumulative reward (value) associated with being in a certain state and following the policy defined by the actor. The actor is responsible for learning and deciding on the optimal policy—the mapping from states to actions. It is essentially the decision-maker or policy function.
• Environment: The environment is the world that the agent interacts with.
• State: The state is the current condition of the environment.
• Action: An action is something that the agent can do in the environment.
• Reward: A reward is a signal that indicates whether an action was good or bad.
• Policy: A policy is a rule that tells the agent which action to take in a given state.
• Value function: A value function is a measure of how beneficial it is to be in a given state.

The basic idea of RL-type algorithms is to improve their policies, and, from this point of view, two main approaches have been developed: on-policy and off-policy learning. On-policy and off-policy are two categories of reinforcement learning algorithms that differ in how they use the collected data for learning. The key distinction lies in whether the learning policy (the policy being optimized) is the same as the policy used to generate the data [12]. In on-policy algorithms, the learning agent follows a specific policy while collecting data, and this policy is the one being improved over time (the data used for learning (experience) come from the same policy that is being updated). In off-policy algorithms, the learning agent has its own exploration policy for the collection of data, but it learns and improves a different target policy (the data used for learning can come from a different (possibly older) policy than the one being updated). Representative examples are SARSA (State–Action–Reward–State–Action) for online learning and Q-learning for offline learning.

RL algorithms for continuous states have evolved significantly over the years, driven by the need to address real-world problems with continuous and high-dimensional state spaces. The early days of RL were dominated by discrete state and action spaces. Dynamic programming algorithms, such as the Bellman equation, were effective in solving problems with small, discrete state spaces. However, they were not suitable for continuous spaces due to the curse of dimensionality. To handle continuous states, researchers introduced function approximation techniques. Value function approximation using methods like tile coding and coarse coding helped to extend RL to continuous state spaces. This approach laid the foundation for the handling of larger and more complex state representations. Policy gradient methods emerged as an alternative to value-based approaches. Instead of estimating the value function, these methods directly learn a parameterized policy. Algorithms like REINFORCE and actor–critic methods became popular for problems with continuous state and action spaces. These algorithms are particularly effective when dealing with high-dimensional and complex problems. Deep Q-networks (DQN) extended RL to problems with high-dimensional state spaces, and later algorithms like Deep Deterministic Policy Gradient (DDPG) and Trust Region Policy Optimization (TRPO) addressed continuous action spaces. These methods leverage neural networks to approximate complex mappings from states to actions [24].

2.2. Deep Deterministic Policy Gradient

The main elements of the DDPG and TD3 algorithms are represented by two types of deep neural networks: the actor neural network and critic neural network. The general structure of these networks is shown in Figure 2 [25], where s—state, a—action and Q—Q-function (critic). The actor is associated with policy-based methods. It can learn a policy that maps states in the environment to actions, trying to find an optimal policy that maximizes the long-term rewards. The actor network receives as input the state of the environment and has as output the action to be applied in this state.

The critic estimates the value functions of the states or of the pairs (state, action) in the environment. The value function indicates the reward that one is expected to receive by starting from a certain state and taking a certain action. The critic evaluates the actions taken by the actor in a given environment and provides feedback regarding how well these actions performed in achieving the desired goals or rewards. The most used technique in establishing the value function is Q-learning. In Q-learning, a function (usually denoted by Q) that associates state–action pairs with the expected value of the future reward is estimated using a deep neural network [26].

In the following, we shall use the usual notations from RL, presented in

Table 1. Notations of main elements of DDPG and TD3 algorithms.

Notation	RL Element
sk	current state
sk+1	next state
ak	current action
ak+1	next action
rk	reward at state sk
Q	Q-function (critic)
π	policy function (actor)
δ	TD target
$\mathbb{A}$	action space
$\mathbb{S}$	state space

.

DDPG is an actor–critic algorithm that presents the advantages of both policy-based and value-based methods and learns optimal estimates for the both policy and value functions. Using the Bellman equation and off-policy data, the Q-function is learnt and then is used to learn the policy [22]. The main idea is the same as in Q-learning: if the optimal Q-function (denoted $Q^{opt}(s,a)$) is known, the optimal action is obtained by solving the following equation [24]:

$$a^{opt}\left(s\right)=arg \underset{a}{\max }{Q}^{opt}(s,a).$$

(1)

The starting point in learning an approximator for $Q^{opt}(s,a)$ is the well-known Bellman equation [12]:

$$Q^{opt}\left(s_k,a_k\right)=\mathbb{E}\left[r_k+\gamma \underset{a\in \mathbb{A}}{\max }{Q}^{opt}\left(s_{k+1},a\right)\right].$$

(2)

Q-learning solves the Bellman optimality equation using the temporal difference (TD) [12]:

$$Q\left(s_k,a_k\right)\leftarrow Q\left(s_k,a_k\right)+\alpha \left[r_k+\gamma \underset{a\in \mathbb{A}}{\max }Q\left(s_{k+1},a\right)-Q\left(s_k,a_k\right)\right],$$

(3)

where

• $\delta =\left[r_k+\gamma \underset{a\in \mathbb{A}}{\max }Q\left(s_{k+1},a\right)-Q\left(s_k,a_k\right)\right]$—TD error;
• $\alpha$—learning rate.

In order to use Q-learning for continuous state and action spaces, neural networks were used as function approximators for Q-values and function policies. So, considering that $Q_w$ is the critic network parametrized by w and ${\pi }_{\theta }$ is the actor network parametrized by θ, relation (3) becomes [12]

$$Q_w\left(s_k,a_k\right)\leftarrow Q_w\left(s_k,a_k\right)+\alpha \left[r_k+\gamma \underset{a\in \mathbb{A}}{\max }{Q}_w\left(s_{k+1},a\right)-Q_w\left(s_k,a_k\right)\right].$$

(4)

In relation (4), the term ${{\phi }_k=r}_k+\gamma \underset{a\in \mathbb{A}}{\max }{Q}_w\left(s_{k+1},a\right)$ represents the target (the desired or goal value that the algorithm is trying to approximate or learn). The target ${\phi }_k$ depends on the parameter w that is to be optimized, so the target varies, and this causes significant difficulties for supervised learning. The solution to this problem in DDPG is to use a new neural network (called the critic target network) that has a similar structure to the critic network but whose parameters do not change rapidly. Moreover, the target ${\phi }_k$ contains the maximization over $a\in \mathbb{A}$, which can be expensive since the Q-value function is a complex network with continuous inputs. To address this problem, a target actor network is used to approximate the optimal policy that maximizes $Q_w\left(s_{k+1},a\right)$. The target networks are denoted in the following by

${\widehat{Q}}_{\widehat{w}}$—target critic network parametrized by $\widehat{w}$;

${\widehat{\pi }}_{\widehat{\theta }}$—target actor network parametrized by $\widehat{\theta }$.

So, the target becomes

$${{\phi }_k=r}_k+\gamma ·{\widehat{Q}}_{\widehat{w}}\left(s_{k+1},{\widehat{\pi }}_{\widehat{\theta }}\left(s_{k+1}\right)\right).$$

(5)

To train the critic $Q_w$, using a minibatch of N samples and a target critic ${\widehat{Q}}_{\widehat{w}}$, we compute

$${{\phi }_i=r}_i+\gamma ·{\widehat{Q}}_{\widehat{w}}(s_{i+1},{\widehat{\pi }}_{\widehat{\theta }}\left(s_{i+1}\right)),$$

(6)

and update w by minimizing the loss function L_C (calculated as the mean square error (MSE) between the current Q-value and the target Q-value):

$$L_c={\displaystyle \frac{1}{N}}{\sum }_i{({\phi }_i-Q_w(s_i, a_i\left)\right)}^2,$$

(7)

In order to update the actor network parameters, the policy gradient is used:

$${\nabla }_{\theta }J={\displaystyle \frac{1}{N}}{\sum }_i\left({\nabla }_a{Q}_w\right(s_i, a_i\left){·\nabla }_{\theta }{\pi }_{\theta }\right(s_i\left)\right),$$

(8)

To update the target network parameters, DDPG performs “soft updates” using Polyak averaging:

$$\begin{array}{c}\widehat{w}\leftarrow \tau ·w+(1-\tau )·\widehat{w}\\ \widehat{\theta }\leftarrow \tau ·\theta +(1-\tau )·\widehat{\theta }\end{array},$$

(9)

where $\tau \in \left[0,1\right],\tau \ll 1$.

In summary, the main steps of the DDPG algorithm are as follows [12]:

• Initialize the critic and actor networks;
• Collect data from the environment and store them in the replay buffer;
• Sample a batch of experiences from the replay buffer;
• Compute the target Q-value using the target networks and update the critic using the mean-squared Bellman error loss;
• Update the actor policy using the sampled batch, aiming to maximize the estimated Q-value;
• Periodically update the target networks with a soft update;
• Repeat steps 3–6 until the desired performance is achieved.

2.3. TD3—The Main Characteristics

The general structure of the TD3 algorithm is presented in Figure 3. It uses six neural networks, namely two critics, two critic targets, an actor, and a corresponding target. During each interaction with the environment, the agent collects experiences in the form of tuples (state, action, reward, next state). Instead of immediately using these experiences to update the policy or value function, the experiences are stored in the replay buffer. The replay buffer is a mechanism used in DDPG and TD3 to store and replay past experiences, promoting more stable and efficient learning in continuous action space reinforcement learning problems. A replay buffer is a key component used to store and sample experiences from the agent’s interactions with the environment. The replay buffer typically has a fixed size, and new experiences overwrite the oldest ones once the buffer is full. This ensures the continuous flow of new experiences while still benefiting from historical data. Training a neural network with sequential experiences can lead to strong correlations between consecutive samples, which may hinder learning. The replay buffer allows for the random sampling of experiences, breaking the temporal correlations and providing more diverse training data.

Overconfidence bias (a term that comes from psychology) defines a disparity between one’s self-assessment of one’s skills and abilities and the actual reality. This phenomenon extends beyond human behavior and is prevalent among RL agents, known in RL terminology as “overestimation bias”. Another drawback of Q-learning algorithms is the possible numerical instability generated when using function approximation to estimate Q-values for continuous or large state–action spaces. Instabilities can arise when training neural networks, especially if they are deep. Issues like vanishing gradients or exploding gradients can affect the stability of the learning process. Consequently, the TD3 algorithm was developed to address these challenges within the actor–critic RL framework, specifically targeting the limitations observed in the DDPG algorithm. TD3 concentrates specifically on the actor–critic framework, implementing three techniques to enhance the DDPG algorithm: (1) clipped double Q-learning, (2) target policy smoothing, and (3) delayed policy and target updates.

Clipped double Q-learning is a means of reducing the overestimation bias in the Q-function. Double Q-learning addresses the overestimation problem by using two sets of Q-values (Q₁ and Q₂), and, during the updates, it uses one set to select the best action and the other set to evaluate this action [23]. Clipped double Q-learning moves a step further by introducing a clipping mechanism during the Q-value updates (as summarized in Figure 4). The idea is to prevent overestimation by limiting the impact of the maximum estimated Q-value.

When updating the Q-values, the algorithm compares the Q-values from the two sets and chooses the smaller one. This clipped value is then used in the update rule. Since only a single actor π is used, a single TD target is then used to update both Q₁ and Q₂ [22].

$${{\phi }_k=r}_k+\gamma ·min\left\{{\widehat{Q}}_{\widehat{w}1}(s_{k+1},{\widehat{\pi }}_{\widehat{\theta }}\left(s_{k+1}\right)), {\widehat{Q}}_{\widehat{w}2}(s_{k+1},{\widehat{\pi }}_{\widehat{\theta }}\left(s_{k+1}\right))\right\}.$$

(10)

Target policy smoothing is a means of making the updates to the policy less aggressive. This helps to prevent the policy from diverging too far from the current policy, which can lead to instability. In order to prevent this, a perturbation $\eta$ (usually chosen as small Gaussian noise) is added to the action of the next state, so that the value evaluation is more accurate. Additionally, the noise itself, as well as the perturbed action, is clipped. The noise is clipped to ensure that it applies to only a small region around the action, while the perturbed action is clipped to ensure that it lies within the range of valid action values.

$${{\phi }_k=r}_k+\gamma ·min\left\{{\widehat{Q}}_{\widehat{w}1}(s_{k+1},{\widehat{\pi }}_{\widehat{\theta }}\left(s_{k+1}\right)), {\widehat{Q}}_{\widehat{w}2}(s_{k+1},{\widehat{\pi }}_{\widehat{\theta }}\left(s_{k+1}\right))\right\}+\eta ,$$

(11)

where $\eta \sim clip(N\left(0,\sigma \right),-c,c)$.

So, the update of w₁ and w₂ (critic parameters) is realized by minimizing two loss functions [22]:

$$\begin{array}{c}{L}_{c1}={\displaystyle \frac{1}{N}}\sum _i{({\phi }_i-Q_{w1}(s_i, a_i\left)\right)}^2\\ L_{c2}={\displaystyle \frac{1}{N}}\sum _i{({\phi }_i-Q_{w2}(s_i, a_i\left)\right)}^2\end{array},$$

(12)

Delayed policy and target updates is a technique by which the target networks are updated less often than the main networks. In TD3, the policy update is delayed. Instead of updating the policy at every time step, the policy network is updated less frequently, typically after a fixed number of iterations or time steps. This delay helps in reducing the correlation between consecutive policy updates, leading to more stable learning and the better exploration of the action space. In addition to delaying the policy updates, TD3 also introduces delayed updates to the target networks (both the actor and the critic networks). In DDPG, the target networks are updated by directly copying the weights from the main networks periodically. However, in TD3, the target networks are updated less frequently than the main networks. Specifically, the target networks are updated less often than the policy network. This delayed updating of the target networks helps in stabilizing the learning process by providing more consistent target values for the temporal difference (TD) error calculation, thus reducing the variance in the update process. These two strategies, delayed policy updates and delayed target updates, work together to improve the stability and performance of the TD3 algorithm, making it more effective in training deep reinforcement learning agents for complex tasks. Similar to DDPG, we use the Polyak averaging technique with the formula below (τ has a very small value) [22]:

$$\begin{array}{c}{\widehat{w}}_1\leftarrow \tau ·w_1+\left(1-\tau \right)·{\widehat{w}}_1,\\ {\widehat{w}}_2\leftarrow \tau ·w_2+(1-\tau )·{\widehat{w}}_2\\ \widehat{\theta } \leftarrow \tau ·\theta +\left(1-\tau \right)·\widehat{\theta } . \end{array},$$

(13)

TD3 is a complex algorithm, but it is relatively easy to implement. Summarizing the above, the Algorithm 1 is as follows [22].

Algorithm 1. TD3
1. Initialize critic networks $Q_{w1}$, $Q_{w2}$ and actor network ${\pi }_{\theta }$ with random parameters $w_1$, $w_2$, $\theta$.    2. Initialize the parameters of the target networks:       ${\widehat{w}}_1{\leftarrow w}_1$, ${\widehat{w}}_2{\leftarrow w}_2$, $\widehat{\theta }\leftarrow \theta$    3. Initialize replay buffer B for k= 1 to T do    4. Select action with exploration noise $a\leftarrow {\pi }_{\theta }+\eta$ and observe reward rk and new state sk+1    5. Store transition (sk, ak, sk+1, rk) in B    6. Sample mini-batch of N transitions (sk, ak, sk+1, rk) from B and compute
$a_k\leftarrow {\widehat{\pi }}_{\widehat{\theta }}\left(s_{i+1}\right)+\eta$    $\eta \leftarrow clip(N\left(0,\sigma \right),-c,c)$    ${{\phi }_k\leftarrow r}_k+\gamma ·min\left\{{\widehat{Q}}_{\widehat{w}1},{\widehat{Q}}_{\widehat{w}2}\right\}+\eta$	(14)
7. Update critic parameters:
$w_1\leftarrow arg \underset{w1}{\min }{\displaystyle \frac{1}{N}}\sum {({\phi }_k-Q_{w1}(s_k,a_k\left)\right)}^2$ $w_2\leftarrow arg \underset{w2}{\min }{\displaystyle \frac{1}{N}}\sum {({\phi }_k-Q_{w2}(s_k,a_k\left)\right)}^2$	(15)
8. Update θ by the deterministic policy gradient
${\nabla }_{\theta }J={\displaystyle \frac{1}{N}}\sum _i\left({\nabla }_a{Q}_{w1}\right(s_k,a_k\left){·\nabla }_{\theta }{\pi }_{\theta }\right(s_k\left)\right)$	(16)
9. Update target networks:
${\widehat{w}}_1\leftarrow \tau ·w_1+\left(1-\tau \right)·{\widehat{w}}_1$    ${\widehat{w}}_2\leftarrow \tau ·w_2+(1-\tau )·{\widehat{w}}_2$    $\widehat{\theta }\leftarrow \tau ·\theta +(1-\tau )·\widehat{\theta }$	(17)
end for

3. Tunning of PID Controllers Using TD3 Algorithm

For PID tuning, TD3 is an excellent choice due to its increased stability, better control over exploration, and superior performance. Compared to DDPG, TD3 offers a higher level of precision and stability. Compared to TRPO, TD3 tends to be simpler to implement and has reduced complexity, making it suitable for PID tuning and other automatic control problems. Regarding its comparison to soft actor–critic (SAC) algorithms, SAC works with stochastic policies, outputting a probability distribution over the actions, while TD3 uses deterministic policies, directly outputting specific actions. Moreover, SAC uses entropy maximization for exploration, automatically balancing exploitation and exploration, while TD3 relies on adding noise to actions, which requires the manual tuning of the exploration parameters.

TD3 reduces the overestimation bias that is common in other reinforcement learning algorithms. TD3 introduces a delay in policy updates, which mitigates the risk of oscillations and instability during learning. The combination of reduced overestimation bias, delayed policy updates, and controlled exploration leads to a stable learning process in TD3. Stability is critical in PID tuning to ensure that the algorithm can converge to a suitable solution without causing fluctuations or inconsistencies in the control parameters. TD3’s approach to mitigating overestimation bias and introducing controlled delays generally leads to higher performance and better convergence rates compared to other methods like DDPG.

In the following, we will consider a standard PID-type controller with the following input–output relation:

$$u=\left[\begin{array}{cc}e& \int edt {\displaystyle \frac{de}{dt}}\end{array}\right]\times {\left[\begin{array}{cc}Kp& Ki Kd\end{array}\right]}^T.$$

(18)

Here,

• u is the output of the actor neural network;
• Kp, Ki, and Kd are the PID controller parameters;
• e(t) = v(t) − y(t), where e(t) is the system error, y(t) is the system output, and v(t) is the reference signal.

The proposed control structure with the tuning of the PID controller parameters using the TD3 algorithm is presented in Figure 5. Given the observations, a TD3 agent decides which action to take using an actor representation. The weights of the actor are, in fact, the gains of the PID controller, so one can model the PID controller as a neural network with one fully connected layer with the error, error integral, and error derivative as inputs (state of the system). The output of the actor neural network represents the command signal (action). The proposed scheme has two time scales: a scale for adapting the PID controller parameters (the weights of the actor) and a scale for analyzing the system’s response to a step input.

The reward can be defined using point-based metrics (e.g., settling time, overshoot, etc.) or metrics based on the system trajectory. In our study, we use a reward function based on the LQG criterion (see Section 5).

4. Tuning PID Controller for Biotechnological System—Classical Approach

Designing a PID controller for biotechnological processes is a difficult task because, in general, these processes are very slow (usually, the time constants of the system are measured in hours); in many cases, they are unstable, which makes practical tuning methods (for example, Ziegler–Nichols-type methods) impossible to apply. For these reasons, to design a PID controller for a biotechnological process, it is necessary first to obtain a linearized model, after which various tuning methods can be applied. We present below the procedure for obtaining a linearized model for a bacterial growth biosystem around an operating point and several tuning methods based on this linearized model.

4.1. Linearization of a Biotechnological System

The modeling of bioprocesses that take place in a fed-batch bioreactor is based on the general mass balance equations [27,28]. Starting from these equations, the general model is represented by a set of nonlinear differential equations of the following form:

$$\dot{\xi }\left(t\right)=f\left(\xi , D, F\right)=\Gamma \cdot \Phi (\xi ,t)-D\xi \left(t\right)+F\left(t\right),$$

(19)

where

• $\xi \left(t\right)\in {\mathfrak{R}}^{n\times 1}$ represents the state vector (the concentrations of the system’s variables);
• $\Phi ={\left[\begin{array}{cccc}{\mu }_1& {\mu }_2& \cdots & {\mu }_m\end{array}\right]}^T$ denotes the vector of the reaction kinetics (the rates of the reactions);
• $\Gamma =\left[{\gamma }_{ij}\right],$ $i=\overline{1,n};j=\overline{1,m}$ is the matrix of the yield coefficients;
• $\Gamma \cdot \Phi (\xi ,t)$ represents the rate of production;
• $-D\xi \left(t\right)+F\left(t\right)$ is the exchange between the bioreactor and the exterior.

This model is strongly nonlinear, a specific characteristic of most biotechnological processes. In order to design a PID controller, the system represented by relation (19) could be linearized around an equilibrium point $\left(\tilde{\xi },\tilde{D},\tilde{F}\right)$ (which is a solution to the equation $f\left(\xi ,D,F\right)=0)$. One obtains the following equations [27]:

$${\displaystyle \frac{d}{dt}}\left(\xi -\tilde{\xi }\right)=A\left(\tilde{\xi }\right)\cdot \left(\xi -\tilde{\xi }\right)-\left(D-\tilde{D}\right)\tilde{\xi }+\left(F-\tilde{F}\right),$$

(20)

with

$$A\left(\tilde{\xi }\right)=\Gamma ·{\left[{\displaystyle \frac{\partial \Phi \left(\xi \right)}{\partial \xi }}\right]}_{\xi =\tilde{\xi }}-\tilde{D}{I}_2,$$

(21)

where $\tilde{\xi }$ represents the equilibrium value of $\xi$, and I₂ represents the second-order unity matrix. The linear model (20)–(21) can be used to tune the PID parameters using classical design formulas or computer-aided design approaches.

In the following, we present the model of the bacterial growth bioprocess that takes place in a continuous stirred-tank bioreactor. The dynamical model of the process can be expressed in the form of a set of differential equations as follows [27]:

$$\begin{array}{c}{\displaystyle \frac{d{\xi }_B}{dt}}=\mu \left({\xi }_S\right)\cdot {\xi }_B-D\cdot {\xi }_B \\ {\displaystyle \frac{d{\xi }_S}{dt}}=-k_1·\mu \left({\xi }_S\right)\cdot {\xi }_B-D\cdot {\xi }_S+F_{in}\end{array},$$

(22)

where ${\xi }_B$ is the biomass concentration, ${\xi }_S$ is the substrate concentration, $F_{in}$ is the input feed rate and $F_{in}=D\cdot S_{in}$ (where $D$ denotes the dilution rate), and $S_{in}$ is the concentration of the influent substrate.

The model can be written in the matrix form

$${\displaystyle \frac{d}{dt}}\left[\begin{array}{c}{\xi }_B\\ {\xi }_S\end{array}\right]=\left[\begin{array}{c}1\\ -k_1\end{array}\right]\cdot \mu ({\xi }_S)\cdot {\xi }_B-D\cdot \left[\begin{array}{c}{\xi }_B\\ {\xi }_S\end{array}\right]+\left[\begin{array}{c}0\\ D\cdot S_{in}\end{array}\right],$$

(23)

and, if one denotes

$$\xi =\left[\begin{array}{c}{\xi }_B\\ {\xi }_S\end{array}\right];\Gamma =\left[\begin{array}{c}1\\ -k_1\end{array}\right];\Phi \left(\xi \right)=\mu \left({\xi }_S\right)\cdot {\xi }_B; F=\left[\begin{array}{c}0\\ D\cdot S_{in}\end{array}\right],$$

(24)

one obtains the form (19).

Because it takes into account the inhibitory effect of the substrate at high concentrations, one of the most used models for the specific growth rate is the Haldane model. The Haldane model is described by the following relation [29]:

$$\mu ({\xi }_S)={\mu }_0·{\displaystyle \frac{{\xi }_S}{K_M+{\xi }_S+{\xi }_S^2/K_S}}$$

(25)

where $K_M$ represents the Michaelis–Menten constant, $K_S$ is the inhibition constant, and ${\mu }_0$ is the maximum specific growth rate.

Since

$${\left[{\displaystyle \frac{\partial \varphi \left(\xi \right)}{\partial {\xi }_B}}\right]}_{\xi =\tilde{\xi }}=\mu \left(\widetilde{{\xi }_S}\right)={\mu }_0·{\displaystyle \frac{\widetilde{{\xi }_S}}{K_M+\widetilde{{\xi }_S}+\widetilde{{\xi }_S^2}/K_S}}$$

and

$${\left[{\displaystyle \frac{\partial \varphi \left(\xi \right)}{\partial {\xi }_S}}\right]}_{\xi =\tilde{\xi }}={\left.{\displaystyle \frac{d\mu \left({\xi }_S\right)}{d{\xi }_S}}\right|}_{\xi =\tilde{\xi }}·{\tilde{\xi }}_B={\mu }_0·{\tilde{\xi }}_B{\displaystyle \frac{K_M-{\tilde{\xi }}_S^2/K_S}{{{(K}_M+{\tilde{\xi }}_S+{\tilde{\xi }}_S^2/K_S)}^2}}\triangleq \beta$$

the linearized matrix has the following form:

$$\mathrm{A}=\left[\begin{array}{cc}\mu \left(\widetilde{{\xi }_S}\right)-\tilde{D}& \beta \\ -k_1\cdot \mu \left(\widetilde{{\xi }_S}\right)& -k_1\cdot \beta -\tilde{D}\end{array}\right].$$

(26)

For the bacterial growth bioprocess, it is important to control the substrate concentration since an overly high value for this concentration leads to the inhibition of biomass growth in the bioreactor [18]. So, one considers ${\xi }_S$ as the controlled variable (output of the system) and the dilution rate D as input. Denoting $y={\xi }_S-\widetilde{{\xi }_S}$ and $u=D-\tilde{D}$, one obtains the standard state-space representation of a linear system:

$$\left\{\begin{array}{c}{\displaystyle \frac{dx}{dt}}=A·x+B·u\\ y=C·x \end{array}\right.,$$

(27)

where

$$\mathrm{A}=\left[\begin{array}{cc}\mu \left(\widetilde{{\xi }_S}\right)-\tilde{D}& \beta \\ -k_1\cdot \mu \left(\widetilde{{\xi }_S}\right)& -k_1\cdot \beta -\tilde{D}\end{array}\right], B=\left[\begin{array}{c}{-\tilde{\xi }}_B\\ {-{\tilde{\xi }}_S+S}_{in}\end{array}\right], C=[0 1].$$

The linearized model (27) can be used to design a PID controller—for example, using the PID Tuner app from Matlab R2021b [19].

4.2. Ziegler–Nichols Method

The Ziegler–Nichols method is a popular heuristic approach for the tuning of PID controllers. It involves the following steps.

Step 1: Set Ki and Kd to zero, leaving only Kp.

Step 2: Increase Kp until the system reaches a critical oscillation (i.e., a steady-state oscillation with a constant amplitude).

Step 3: Record the value of Kp when the system starts oscillating, which is called the ultimate gain (Ku), and the period of the oscillation (Pu).

Step 4: Use the following formulas to calculate the PID parameters:

P controller: Kp = 0.5 × Ku;

PI controller: Kp = 0.45 × Ku, Ki = 1.2 × Kp/Pu;

PID controller: Kp = 0.6 × Ku, Ki = 2 × Kp/Pu, Kd = Kp × Pu/8.

This method is effective for systems with well-defined oscillation behavior but may not be suitable for highly sensitive systems or those with noise.

4.3. Pole Placement Method

The pole placement method is a classical control design technique used to place the poles of a system in specific locations in the complex plane, thereby determining the system’s desired behavior (such as stability, damping, and natural frequency). In the context of second-order systems, this method allows us to control key performance characteristics such as overshoot, the settling time, and the damping ratio by appropriately selecting the poles of the system.

The PID controller parameters were tuned using the pole placement method for a second-order system described in [5]. If the process transfer function has the following form

$$P(s)={\displaystyle \frac{b_{1}s+b_2}{s^2+a_{1}s+a_2}},$$

(28)

the closed-loop system is of the third order, and the characteristic polynomial is

$$s\left(s^2+a_{1}s+a_2\right)+(b_{1}s+b_2)(K_D{s}^2+K_{P}s+K_I),$$

(29)

Choosing the closed-loop characteristic equation of a third-order system is achieved as follows:

$${(s+\alpha {\omega }_0)(s}^2+2\xi {\omega }_{0}s+{\omega }_0^2),$$

(30)

with $\alpha , {\omega }_0, \xi$ as the tuning parameters.

Equating coefficients of equal power in s in Equations (29) and (30), one obtains a system of three equations, from which one obtains $K_p, K_i, K_d$:

$$K_p={\displaystyle \frac{a_2{b}_2^2-a_2{b_{1}b}_2\left(\alpha +2\xi ){\omega }_0\right)-\left({b_2-a_{1}b}_1\right)(b_2\left(1+2\alpha \xi \right){\omega }_0^2+\alpha b_1{\omega }_0^3}{b_2^3-{b_{1}b}_2^2\left(\alpha +2\xi \right){\omega }_0+b_1^2{b}_2\left(1+2\alpha \xi \right){\omega }_0^2-{\alpha b}_1^3{\omega }_0^3}}$$

$$K_i={\displaystyle \frac{(-a_1{b_{1}b}_2+a_2{b}_1^2+b_2^2)\alpha {\omega }_0^3}{b_2^3-{b_{1}b}_2^2\left(\alpha +2\xi \right){\omega }_0+b_1^2{b}_2\left(1+2\alpha \xi \right){\omega }_0^2-{\alpha b}_1^3{\omega }_0^3}}$$

(31)

$$K_d={\displaystyle \frac{{-a}_1{b}_2^2+a_2{b_{1}b}_2+b_2^2(\alpha +2\xi ){\omega }_0-b_1{b}_2{\omega }_0^2\left(1+2\alpha \xi \right)+b_1^2\alpha {\omega }_0^3}{b_2^3-{b_{1}b}_2^2\left(\alpha +2\xi \right){\omega }_0+b_1^2{b}_2\left(1+2\alpha \xi \right){\omega }_0^2-{\alpha b}_1^3{\omega }_0^3}}$$

5. Simulation Results

The tuning approaches presented in Section 3 and Section 4 were implemented in the Matlab/Simulink environment. The Simulink implementation of the bacterial growth bioprocess is presented in Figure 6. The bioprocess parameters used in the simulations were taken from reference [29]:

$${\mu }_0=6 {\mathrm{h}}^{-1},K_M={\displaystyle \frac{10 \mathrm{g}}{l}},K_S={\displaystyle \frac{100 \mathrm{g}}{l}}, k_1=1, S_{in}={\displaystyle \frac{100 \mathrm{g}}{l}}$$

5.1. RL Approach

The tuning of the PID controller parameters using the TD3 algorithm is performed offline using a mathematical model of the controlled process. The Matlab implementation of the PID tuning structure shown in Figure 5 is presented in Figure 7 and is based on the RL-TD3 agent implemented in Matlab/Simulink [30].

The observation block has the three elements $\left[P\left(t_n\right)I\left(t_n\right)D\left(t_n\right)\right]$ (see Figure 8), while the reward block implements the formula for the reward. We used a reward function based on the LQG criterion:

$$J=\underset{T\to \mathrm{\infty }}{\lim }\left({\displaystyle \frac{1}{T}}{\int }_0^T({a·\left(ref-y\right)}^2+\mathrm{b}·u^2(t\left)\right)dt\right),$$

(32)

with $a,b$ being two weighting coefficients (see Matlab implementation in Figure 9).

The parameter a corresponds to penalties on state deviations. Larger values of a indicate the stricter control of the corresponding states. It is chosen as the inverse of the square of the maximum acceptable deviation for each state. The b parameter penalizes control efforts. Higher values for this parameter lead to more conservative control actions.

The training parameters for the actor and critic networks are crucial in ensuring stable and efficient learning. They are often tuned through experimentation and hyperparameter optimization techniques like grid search or random search. The goal is to balance stability, the convergence speed, and the accuracy of the Q-value estimates. Below is a detailed breakdown of the key training parameters for both the actor and critic networks. The experience buffer (or replay buffer) is a critical component of TD3 and other off-policy reinforcement learning algorithms. It stores past experiences collected by the agent and allows them to be reused for training. Common sizes range from 10⁵ to 10⁶. The mini-batch size is the number of experiences sampled from the experience buffer for each training step. Common batch sizes range from 64 to 256. The discount factor (γ) is usually set between 0.95 and 0.99. Higher values (closer to 1) prioritize long-term rewards, while lower values focus on immediate rewards. The learning rates (for the actor and critic) ensure stable policy updates. Common values range from 10⁻² to 10⁻⁴. An episode in TD3 training is a complete sequence of interactions between the agent and the environment, from an initial state to termination. Episodes are used to collect experiences for training, evaluate the agent’s performance, and facilitate exploration. The length of an episode depends on the environment and the agent’s behavior, and multiple episodes are typically required to train the agent effectively. The Adam optimizer (short for Adaptive Moment Estimation) is a popular optimization algorithm used in training neural networks and other machine learning models. It combines the benefits of two other optimization techniques: the Adaptive Gradient Algorithm (AdaGrad) and Root Mean Square Propagation (RMSProp). Adam is known for its efficiency, robustness, and ability to handle sparse gradients, making it the default choice for many deep learning tasks, including reinforcement learning algorithms. The parameters that are utilized for the actor and critic networks of the TD3 agent are presented in

Table 2. Training parameters for actor and critic networks.

Parameter	Value
Mini-batch size	128
Experience buffer length	500,000
Gaussian noise variance	0.1
Steps in episode	200
Maximum number of episodes	1000
Optimizer	Adam
Discount factor	0.97
Fully connected learning size	32
Critic learning rate	0.01
Reward function	a = 0.1, b = 0.02
Actor learning rate	0.01

.

The training process is presented in Figure 10.

It can be seen that the algorithm converges after approximately 100 episodes. The gains of the PID controller are the absolute weights of the actor representation. After 200 episodes, the following values of the PID controller were obtained:

$$K_P=0.62;K_I=3.57;K_D=0.00023$$

With the gains obtained from the RL agent, a step response simulation was performed. The time evolution of the output of the system (substrate concentration) is presented in Figure 11, and the command signal and the biomass concentration are presented in Figure 12.

5.2. Classical Approach

The nonlinear bioprocess was linearized around the equilibrium point $\tilde{D}=3.6 {\mathrm{h}}^{-1},\widetilde{{\xi }_B}=80 \mathrm{g}/\mathrm{L},\widetilde{{\xi }_S}=23 \mathrm{g}/\mathrm{L}$. One obtains the linear model

$$\left\{\begin{array}{c}{\displaystyle \frac{dx}{dt}}=A·x+B·u\hfill \\ y=C·x\hfill \end{array}\right.,$$

(33)

with

$$\mathrm{A}=\left[\begin{array}{cc}0& 59.04\\ -3.60& -62.65\end{array}\right] B=\left[\begin{array}{c}-80\\ 77\end{array}\right], C=[0 1]$$

The PID controller parameters were tuned using the PID Tuner app from Matlab (see Figure 13). The values of the PID controller are

$$K_P=0.207;K_I=28.64;K_D=0.00037$$

The time evolution of the output of the system (substrate concentration) is presented in Figure 14, and the command signal and the biomass concentration are presented in Figure 15.

From Figure 12 and Figure 15, it can be seen that the control of the substrate is very important (too high a value inhibits the concentration of biomass), with the main purpose of the regulation system being an increase in biomass production.

For the linear system (33), the two tuning methods presented in Section 4.2 and Section 4.3 were also applied: the Ziegler–Nichols and pole placement methods. The main results and step response characteristics are summarized in

Table 3. Tuned parameters and step response characteristics.

Tuning Method	Kp	Ki	Kd	Overshoot [%]	Settling Time [h]
TD3-based tuning	0.62	3.57	0.00023	0	0.8
PID Tuner app from Matlab	0.207	28.64	0.00037	20–40%	0.4–0.5
Ziegler–Nichols tuning	1.76	11.36	0.00054	35–45%	0.5–0.6
Pole placement tuning	1.32	14.27	0.00043	10–22%	0.5–0.55

.

The response obtained through the classical approaches is more aggressive, with a large overshoot and a slightly shorter response time. The response in the case of TD3-based tuning is slower and it has no overshoot.

6. Conclusions

In this paper, the use of reinforcement learning techniques for the tuning of PID controllers was proposed. The proposed method is very useful from a practical point of view because many industrial processes still use PID controllers, and their tuning is a particularly important step. The proposed tuning method is based on the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and is applied to determine the parameters of PID controllers used in the control of continuous nonlinear systems. The RL-based tuning method is compared with classical techniques that are based on the linearization of the nonlinear system around an operating point. In the first case, the tuning of the controller is performed using the TD3 algorithm, which presents a series of advantages compared to other similar RL approaches dedicated to the control of continuous systems. This algorithm is an off-policy actor–critic-based method, and it was chosen as it does not require a system model and works in environments with continuous action and state spaces. In the classical approach, the nonlinear system was linearized around an equilibrium point and then standard tuning methods were used to determine the PID parameters. The presented techniques were applied to the control of a biotechnological system—a bacterial growth process—that took place in a fed-batch bioreactor. The simulations demonstrate the possibility of using machine learning algorithms to tune the parameters of PID controllers with the aim of controlling nonlinear systems.