Soft Actor-Critic Reinforcement Learning Improves Distillation Column Internals Design Optimization

Abstract

Amid the advancements in computer-based chemical process modeling and simulation packages used in commercial applications aimed at accelerating chemical process design and analysis, there are still certain tasks in design optimization, such as distillation column internals design, that become bottlenecks due to inherent limitations in such software packages. This work demonstrates the use of soft actor-critic (SAC) reinforcement learning (RL) in automating the task of determining the optimal design of trayed multistage distillation columns. The design environment was created using the AspenPlus^® software (version 12, Aspen Technology Inc., Bedford, Massachusetts, USA) with its RadFrac module for the required rigorous modeling of the column internals. The RL computational work was achieved by developing a Python package that allows interfacing with AspenPlus^® and by implementing in OpenAI’s Gymnasium module (version 1.0.0, OpenAI Inc., San Francisco, California, USA) the learning space for the state and action variables. The results evidently show that (1) SAC RL works as an automation approach for the design of distillation column internals, (2) the reward scheme in the SAC model significantly affects SAC performance, (3) column diameter is a significant constraint in achieving column internals design specifications in flooding, and (4) SAC hyperparameters have varying effects on SAC performance. SAC RL can be implemented as a one-shot learning model that can significantly improve the design of multistage distillation column internals by automating the optimization process.

1. Introduction

Computer-aided chemical process simulation has been an important tool not just in chemical engineering education, but also in commercial platforms for the design and analysis of industrial systems that process bulk chemicals, specialty chemicals, pharmaceuticals, and many more [1,2]. Among several software packages, AspenPlus^®, which is part of Aspen Tech’s suite of software packages [3,4,5], has been the state of the art due to its wide array of functional capabilities (e.g., property analysis, rigorous models, process flowsheeting, API for external software such as Fortran and Python, etc.) and its extensive database of chemical and physical properties [3,6,7,8,9,10]. Despite being the most advanced chemical process modeling software, this software has its inherent limitations [11]. One such limitation is that it still requires users to perform manual iterative optimization of certain designs, which can sometimes result in extended periods spent on iterations by the user or in a sub-optimal final design [12]. These challenges can be exacerbated when dealing with a high-dimensional space in the design parameters being explored [11,12]. Addressing these challenges and establishing solutions can make the design process via this software tool (and other similar software packages) significantly more efficient. This work aimed to demonstrate the use of reinforcement learning (RL), specifically the soft actor-critic (SAC) algorithm [13,14], to improve the design optimization task for the internals of multistage distillation columns (Figure 1).

1.1. The Challenge: Distillation Column Internals Design

The specification of column internals in a multistage distillation column significantly dictates the hydraulics involved in the interactions of the vapor and liquid streams through the various stages inside the column (Figure 1). This consequently creates a feedback loop in the usual design process—the hydraulics in the column become metrics of the goodness of the column internals design specifications [15]. In this study, the hydraulics parameter called the percent (%) approach to flooding, which is denoted here using the notation $f_{\%}$, is used to measure the goodness of the distillation column internals design. This parameter $f_{\%}$ is not the only metric of good column design [2,15], but it is a significant indicator [16]. The numerous nonlinear relations and models involved in computing the column internals parameters that produce allowable values of $f_{\%}$ make this design task very difficult to automate using traditional programming techniques. Even the AspenPlus^® software does not currently have a built-in automated optimization function for this task [3,10]. Hence, this study demonstrates an SAC RL approach to provide this valuable automation. For a seamless integration between the distillation column internals parameters and the SAC RL model, this study uses the RadFrac module available in AspenPlus^®. The RadFrac module is the most rigorous module in AspenPlus^® fitting to be used in column internals design because of the stage-by-stage vapor–liquid calculations, hydraulics analysis, and its support for trayed and packed column designs [17,18]. A Python (version 3.9.21, Python Software Foundation, Wilmington, Delaware, USA) module called ‘pywin32’ [19] serves as the API to read data from and send data to AspenPlus^®.

1.2. The Solution: SAC Algorithm-Based Design Optimization

This work evaluates the capability of SAC to automatically learn the best design parameter settings in the RadFrac column. SAC is an algorithm that optimizes a stochastic policy in an off-policy manner, which implements the combined capabilities of stochastic policy optimization and deep deterministic policy gradient (DDPG) approaches [13,14,20]. Because of the blend of stochastic and deterministic capabilities, the prediction of the AC model for the next action steps are well within the bounds of the defined action space [13,14]. This is a feature that can be fitting to implement in distillation column design because the column parameters are allowed to be sampled from only a finite range of values due to limitations in fluid dynamics, vapor–liquid equilibrium, energy balance, and other constraints imposed by natural law.

Note that a DDPG model was also evaluated during the preliminary works for this project, and the DDPG model predictions failed to stay within the bounds of the action variables (column internals design parameters), resulting in input value errors of the column design parameters when implemented in the AspenPlus^® model file. This prompted the researchers to eliminate DDPG and focus only on SAC as the RL to evaluate for column design optimization in this work.

2. Methodology

2.1. Environment: Multistage Distillation Column as RadFrac in AspenPlus^®

A model of tray-type multistage distillation for the separation of ethanol (C₂H₅OH) and water (H₂O) was chosen because this binary chemical system is commonly used in many design analysis tasks in chemical engineering. Nonetheless, the methodology of this study should naturally be applied to other chemical systems undergoing distillation in a trayed column. The well-established AspenPlus^® software was chosen as the modeling platform for the tray-type column using its RadFrac module. Please refer to the accompanying AspenPlus^® file (.apw and bkp) for all other details of the model used (see Data Availability section).

The details of the binary mixture to be separated via distillation are based on the literature materials [2,15] and are as follows. A mixture of ethanol and water must be distilled in a sieve tray-type distillation column. The feed contains 35% ethanol and 65% water, and the distillate product stream must contain at least 80% ethanol. Note that 87% ethanol is the azeotropic point of the binary system under atmospheric pressure [15], which is the reason for the target distillate concentration of ethanol. The feed is flowing at 20,000 kmol/h, is a saturated liquid, and is coming to the column at pressure of 1 bar. The distillate rate must be at 104 kmol/h and the external reflux ratio must be 2.5. After some preliminary graphical methods for estimating the number of stages (using the McCabe–Thiele method) [2,15,21], it was determined that the total number of stages must be 20 and the feed tray must be on stage 10. The column must have a total condenser and a partial reboiler. The task is to design a sieve tray type multistage distillation column (see Figure 2) using the RadFrac module in AspenPlus^®. Specifically, the column internals must be optimized with the objective of achieving the approach to flooding $f_{\%}$ levels to within the acceptable range, e.g., 80–90% (see Section 2.2).

To simplify the modeling in AspenPlus^® and the implementation of SAC RL, the feed tray was assigned to the TOP section of the column. So, the TOP section covers Stage 1 to Stage 10 while the BOT section covers Stage 11 to Stage 20.

2.2. Distillation Column Flooding

The state variable for this study is the column flooding measured as % approach to flooding (denoted here as $f_{\%}$), which is one of the main performance metrics for sizing distillation column internals [16]. Flooding is the excessive accumulation of liquid in the column, resulting in poor equilibrium contact between the vapor and liquid streams. The % approach to flooding must be minimized, and a typical range of % approach to flooding is 65% to 90% according to Wankat [15], while Jones and Mellborn [22] suggest a 75% approach to flooding is a good value for various chemical systems. There are four main mechanisms of flooding, as shown in Figure 3. Downcomer backup flooding (Figure 3A) occurs when the liquid is backed up into the downcomer due to tray pressure drop, which is usually caused by short tray spacing [16,22]. Downcomer choke flooding (Figure 3B) occurs when the entrance to the downcomer is too narrow, resulting in build-up of excess friction losses for the liquid to overcome [16]. Spray entrainment flooding (Figure 3C) occurs when there is a relatively low amount of liquid on the tray because of relatively short weir height and narrow downcomer clearance [16]. Froth entrainment flooding (Figure 3D) occurs when significant height of froth reaches the tray above, resulting in compromise of the stream concentrations, and this can be a significant effect of settings in the tray spacing, sieve hole diameter, and fraction of the tray allocated for vapor–liquid exchange [15,16].

2.3. Notation

We now present notations to formally define the various components of the task in relation to the established notations in RL. The RL problem at hand is a policy search in a Markov Decision Process (MDP) involving the tuple $\left(S,A,p,r\right)$. The state space $S$ is a continuous space that consists of the following column internals design parameters (Figure 1 and Figure 2): top section (TOP) % approach to flooding and bottom section (BOT) % approach to flooding. Note that the state % approach to flooding can be greater than 100% according to distillation column design principles, so $S \to [0,\infty )$. The action space $A$ is a continuous space that consists of the following column internals design parameters (Figure 1 and Figure 2): tray spacing, weir height, downcomer clearance, weir side length, and hole diameter. The action space is bounded by the allowable values of column design parameters in the AspenPlus^® software, and the actual values also scale with the units (i.e., ft, cm, or mm; kg or lb; etc.) they are assigned in the software by the user, as explained above. The state transition probability $p$ represents the probability density of the next state $s_{t+1}\in S$ given the current state $s_t\in S$ and action $a_t\in A$. The environment (i.e., column internals design) emits a reward $r\in R \to [r_{min}, r_{max}]$ from each transition (i.e., process simulation). Note that this study also evaluated how various forms of the reward system $R$ may affect the SAC performance.

Table 1. Summary of action and state variables used in the study.

Action/State Variable	Description
$\mathrm{Action} 1 (A_1$) = TOP Downcomer clearance	Distance between the bottom edge of the downcomer and the tray below; value range: [30, 150]; units: mm
$\mathrm{Action} 2 (A_2$) = TOP Tray spacing	Distance between two consecutive trays; value range: [1, 7]; units: ft
$\mathrm{Action} 3 (A_3$) = TOP Weir height	Height of a tray outlet weir, which regulates the amount of liquid build-up on the plate surface; value range: [10, 150]; units: mm
$\mathrm{Action} 4 (A_4$) = TOP Sieve hole diameter	Diameter of the holes on the sieve tray; value range: [5, 15]; units: mm
$\mathrm{Action} 5 (A_5$) = TOP Weir side length	Length of the tray outlet weir; value range: [0.1, 1]; units: ft
$\mathrm{Action} 6 (A_6$) = BOT Downcomer clearance	Distance between the bottom edge of the downcomer and the tray below; value range: [30, 150]; units: mm
$\mathrm{Action} 7 (A_7$) = BOT Tray spacing	Distance between two consecutive trays; value range: [1, 7]; units: ft
$\mathrm{Action} 8 (A_8$) = BOT Weir height	Height of a tray outlet weir, which regulates the amount of liquid build-up on the plate surface; value range: [10, 150]; units: mm
$\mathrm{Action} 9 (A_9$) = BOT Sieve hole diameter	Diameter of the holes on the sieve tray; value range: [5, 15]; units: mm
$\mathrm{Action} 10 (A_{10}$) = BOT Weir side length	Length of the tray outlet weir; value range: [0.1, 1]; units: ft
$\mathrm{Action} 11 (A_{11}$) = Column diameter **	Diameter of the column; value range: [4, 9]; units: ft
$\mathrm{State} 1 (S_1$$)=\mathrm{TOP} f_{\%}$	$\mathrm{Flooding} f_{\%}$ $\mathrm{of} \mathrm{the} \mathrm{column} \mathrm{top} \mathrm{section}; \mathrm{value} \mathrm{range}: [0,\infty )$; units: %
$\mathrm{State} 2 (S_2$$)=\mathrm{BOT} f_{\%}$	$\mathrm{Flooding} f_{\%}$ $\mathrm{of} \mathrm{the} \mathrm{column} \mathrm{bottom} \mathrm{section}; \mathrm{value} \mathrm{range}: [0,\infty )$; units: %

** Note: column diameter $A_{11}$ was included in the SACR RL action space (i.e., varied) only when testing the effect of column diameter (see Section 3.3). Otherwise, $A_{11}$ was fixed when SAC RL was learning to optimize the other column design parameters.

summarizes the variables for the action space and state space with their corresponding descriptions according to the actual settings of the AspenPlus^® model used in the study. Also refer to Figure 2 for visualization of the action variables defined in Table 1. The action space $A$ is multidimensional and consists of the column design parameters, denoted as $A_1, A_2, \dots A_{11}$, as shown in Table 1. The state space $S$ is two-dimensional and consists of $S_1$ and $S_2$ (Table 1). Note that each section of the column, i.e., TOP and BOT, is represented in the action and state spaces even though the design parameters between the sections are similar.

2.4. SAC RL

We present in this section the key equations of the SAC RL algorithm, but the reader should consult the original works [13,14,20] that established this algorithm if more details are needed.

The goal in SAC RL is to determine a policy $\pi$, which maximizes two components: (1) the expected return $R\left(\tau \right)$ from the rewards $r_t$, where $\tau =\left(s_0, a_0, s_1, a_1, \dots \right)$ is the trajectory, when the agent acts according to the policy and (2) the entropy of the policy $H\left(\pi \right)$, i.e., entropy of the predicted next actions. To achieve this, the policy model $\pi$ must be estimated by an optimal policy ${\pi }^{*}$ that is calibrated (via its model parameters) using data collected about the environment [14]. Policy, in this context, is a rule used to decide what next actions to take based on the current state of the environment. Specifically, $\pi$ is a probability distribution over the possible actions $a_t$ given the current state $s_t$ of the environment, i.e., $a_t~\pi \left(·|s_t\right)$. The optimal policy ${\pi }_{\theta }^{*}$ is modeled using deep neural network with network weights $\theta$, and this becomes the actor network. In essence, the main goal in training an SAC RL model can then be expressed generally as follows:

$${\pi }^{\mathrm{*}}={\pi }^{\mathrm{*}}=\arg \underset{\pi }{\max }{\mathsf{{\rm E}}}_{\tau ~\pi }\left[{\sum }_{t=0}^{\mathrm{\infty }}{\gamma }^t\left(R\left(s_t,a_t,s_{t+1}\right)+\alpha H\left(\pi \left(·|s_t\right)\right)\right)\right]$$

(1)

where $\mathsf{{\rm E}}$ is the expectation, $\alpha$ is the trade-off coefficient for the entropy $H$ term, and $\gamma$ is the discount factor in the value function [20].

In general, the reward function $R$ depends on the current state $s_t$, the action just taken $a_t$, and the next state $s_{t+1}$, such that $r_t=R\left(s_t,a_t,s_{t+1}\right)$. The dependence in the function $R$ is frequently simplified to have dependence on the state only, i.e., $r_t=R\left(s_t\right)$, or on the state action only, i.e., $r_t=R\left(s_t,a_t\right)$. The SAC model used in this study used the former model $r_t=R\left(s_t\right)$. One of the two components of the objective function in the SAC RL agent is to maximize the cumulative reward over a trajectory $\tau$, and this cumulative reward is commonly called the return $R\left(\tau \right)$. It has been shown that a discounted return helps in the convergence of RL models, so this is also adopted in the SAC learning model such that $R\left(\tau \right)=\sum _{t=0}^{\infty }{\gamma }^t{r}_t$, where $\gamma$ is the discount factor [14].

The maximum entropy term $H\left(\pi \right)$ (second term in Equation (1)) is the unique feature of the SAC RL. By design, this term encourages the RL model to still explore as it learns to prioritize actions with projected high rewards [14]. This prevents the model from being focused on sub-optimal actions [23].

The determination of the best actor network ${\pi }_{\theta }^{*}$ is achieved by using value feedback from a system of critic networks (see Figure 1), which are also deep neural networks and that implement double Q-learning [14]. SAC concurrently learns the policy ${\pi }^{*}$ and two Q-functions $Q_{{\varphi }_1}$ and $Q_{{\varphi }_2}$ of the critic, where ${\varphi }_1$ and ${\varphi }_2$ are the network weights of the critic networks.

The smoothing constant, $\tau \in [0, 1]$, is a hyperparameter that controls updating the weights of the target networks for the Q-functions $Q_{{{\varphi }_1}^{\prime }}$ and $Q_{{{\varphi }_2}^{\prime }}$ [24]. In notation: ${{\varphi }_1}^{\prime }\leftarrow {\tau \varphi }_1+(1-\tau ){{\varphi }_1}^{\prime }$ and ${{\varphi }_2}^{\prime }\leftarrow {\tau \varphi }_2+(1-\tau ){{\varphi }_2}^{\prime }$.

The fourth hyperparameter to be evaluated is the “replay buffer length”, which originates from using replay buffer as a solution to extract more information from the history sequential data for RL training [24]. Replay buffer is a component of the computation of the target Q-functions $Q_{{{\varphi }_1}^{\prime }}$ and $Q_{{{\varphi }_2}^{\prime }}$ presented above. The replay buffer is a finite-sized (replay buffer length) memory datum. The buffer is created by sampling transitions from the environment according to the exploration policy, and the tuple $\left(s_t,a_t,r_t,s_{t+1}\right)$ is stored in the replay buffer. The buffer eventually is filled as more tuple entries are added. When the buffer length is reached, the oldest samples are discarded while maintaining the buffer length. At each timestep, the actor and critic are updated by sampling a minibatch uniformly from the buffer.

2.5. Implementation: OpenAI Gymnasium, PyTorch (Version 2.6.0+cu118)

To eliminate the need to create our custom RL environment, we used the prebuilt RL module called “Gymnasium” by OpenAI [25,26]. This module implements the necessary algorithms and functions to define variable spaces and efficient action space-sampling mechanisms to ensure that RL models are exposed to a wide variety of scenarios, resulting in improved learning performance [25,27]. The Python codes are based on prior work with code implementation of RL policy gradient methods [28].

2.5.1. Action Variables

Since all action variables in $A$ are continuous, these were defined using the ‘Gymnasium.Box()’ function, which is designed to handle a tensor of continuous variables. The lower and upper bounds of these action variables were required arguments in the ‘Gymnasium.Box()’ functions.

2.5.2. State Variables

Defining the state space was very similar to how the action space (Section 2.5.1) was defined using the ‘Gymnasium.Box()’ function.

2.5.3. Reward Scheme

The reward schemes studied are summarized in

Table 2. Summary of reward schemes used in this study.

Reward Model	Definition
Scheme 1	$r={\displaystyle \sum _{k\in \left\{\mathrm{TOP}, \mathrm{BOT}\right\}}^K}{r}_k=r_{TOP}+r_{\mathrm{BOT}}$ $\mathrm{where} r_k$ $\mathrm{is} \mathrm{the} \mathrm{reward} \mathrm{in} \mathrm{section} k$ $\mathrm{of} \mathrm{the} \mathrm{column}. \mathrm{For} \mathrm{each} k$: $r_k=\left\{\begin{array}{cc}\hfill 100,& {80\le f}_{\%}<90\hfill \\ \hfill 100-2d_{err},& f_{\%}<80\hfill \\ \hfill 100-2d_{err},& f_{\%}\ge 90\hfill \end{array}\right.$ $\mathrm{where} d_{err}=\left|f_{\%}-80\right|$ $\mathrm{or} d_{err}=\left|f_{\%}-90\right|$
Scheme 2	$r={\displaystyle \sum _{k\in \left\{\mathrm{TOP}, \mathrm{BOT}\right\}}^K}{\displaystyle \frac{1}{K}}{r}_k=0.5r_{TOP}+0.5r_{\mathrm{BOT}}$ $\mathrm{where} r_k$ $\mathrm{is} \mathrm{the} \mathrm{reward} \mathrm{in} \mathrm{section} k$ $\mathrm{of} \mathrm{the} \mathrm{column}. \mathrm{For} \mathrm{each} k$: $r_k=\left\{\begin{array}{c} 100, f_{\%}\le 85\\ -d_{err}, f_{\%}>85\end{array}\right.$ $\mathrm{where} d_{err}=\left|f_{\%}-85\right|$
Scheme 3	$\mathrm{Reward} r_k$ $\mathrm{is} \mathrm{based} \mathrm{on} \mathrm{the} \mathrm{intervals} \mathrm{of} f_{\%}$ $\mathrm{with} \mathrm{highest} \mathrm{score} \mathrm{for} \mathrm{the} \mathrm{target} \mathrm{interval} \left[80, 90\right)$. $r={\displaystyle \sum _{k\in \left\{\mathrm{TOP}, \mathrm{BOT}\right\}}^K}{\displaystyle \frac{1}{K}}{r}_k=0.5r_{TOP}+0.5r_{\mathrm{BOT}}$ $\mathrm{where} r_k$ $\mathrm{is} \mathrm{the} \mathrm{reward} \mathrm{in} \mathrm{section} k$ $\mathrm{of} \mathrm{the} \mathrm{column}. \mathrm{For} \mathrm{each} k$: $r_k=-100$ $\mathrm{if} f_{\%}\in \left[200, \infty \right)$ $; r_k=-80$ $\mathrm{if} f_{\%}\in \left[180, 200\right)$ $; r_k=-60$ $\mathrm{if} f_{\%}\in \left[160, 180\right)$ $; r_k=-40$ $\mathrm{if} f_{\%}\in \left[140, 160\right)$ $; r_k=-30$ $\mathrm{if} f_{\%}\in \left[120, 140\right)$ $; r_k=-20$ $\mathrm{if} f_{\%}\in \left[100, 120\right)$ $; r_k=-10$ $\mathrm{if} f_{\%}\in \left[90, 100\right)$ $; r_k=0$ $\mathrm{if} f_{\%}\in \left[0, 400\right)$ $; r_k=20$ $\mathrm{if} f_{\%}\in \left[40, 50\right)$ $; r_k=40$ $\mathrm{if} f_{\%}\in \left[50, 60\right)$ $; r_k=60$ $\mathrm{if} f_{\%}\in \left[60, 70\right)$ $; r_k=80$ $\mathrm{if} f_{\%}\in \left[70, 80\right)$ $; r_k=100$ $\mathrm{if} f_{\%}\in \left[80, 90\right)$
Scheme 4	$\mathrm{Reward} r_k$ is similar to Scheme 3 but with reward values lower by a factor of 10: $r={\displaystyle \sum _{k\in \left\{\mathrm{TOP}, \mathrm{BOT}\right\}}^K}{\displaystyle \frac{1}{K}}{r}_k=0.5r_{TOP}+0.5r_{\mathrm{BOT}}$ $\mathrm{where} r_k$ $\mathrm{is} \mathrm{the} \mathrm{reward} \mathrm{in} \mathrm{section} k$ $\mathrm{of} \mathrm{the} \mathrm{column}. \mathrm{For} \mathrm{each} k$: $r_k=-10$ $\mathrm{if} f_{\%}\in \left[200, \infty \right)$ $; r_k=-8$ $\mathrm{if} f_{\%}\in \left[180, 200\right)$ $; r_k=-6$ $\mathrm{if} f_{\%}\in \left[160, 180\right)$ $; r_k=-4$ $\mathrm{if} f_{\%}\in \left[140, 160\right)$ $; r_k=-3$ $\mathrm{if} f_{\%}\in \left[120, 140\right)$ $; r_k=-2$ $\mathrm{if} f_{\%}\in \left[100, 120\right)$ $; r_k=-1$ $\mathrm{if} f_{\%}\in \left[90, 100\right)$ $; r_k=0$ $\mathrm{if} f_{\%}\in \left[0, 40\right)$ $; r_k=2$ $\mathrm{if} f_{\%}\in \left[40, 50\right)$ $; r_k=4$ $\mathrm{if} f_{\%}\in \left[50, 60\right)$ $; r_k=6$ $\mathrm{if} f_{\%}\in \left[60, 70\right)$ $; r_k=8$ $\mathrm{if} f_{\%}\in \left[70, 80\right)$ $; r_k=10$ $\mathrm{if} f_{\%}\in \left[80, 90\right)$
Scheme 5	The reward is built on a binary baseline scheme: $r={\displaystyle \sum _{k\in \left\{\mathrm{TOP}, \mathrm{BOT}\right\}}^K}{\displaystyle \frac{1}{K}}{r}_k=0.5r_{TOP}+0.5r_{\mathrm{BOT}}$ $\mathrm{where} r_k$ $\mathrm{is} \mathrm{the} \mathrm{reward} \mathrm{in} \mathrm{section} k$ $\mathrm{of} \mathrm{the} \mathrm{column}. \mathrm{For} \mathrm{each} k$: $r_k=\left\{\begin{array}{c} 1, f_{\%}\le 85\\ -1, f_{\%}>85\end{array}\right.$

. Reward is based on the flooding level $f_{\%}$ of each column section and the scheme for awarding the reward values. Previous works have demonstrated that the reward scheme can significantly affect the performance of RL algorithms [29,30,31,32]. The unique aspect of the reward schemes adopted for this study is the separate computation of reward for each column section $r_k$ and then the aggregation of these by summation (weighted and unweighted) into a single reward value $r$ (Table 2). Schemes 1 and 2 are built on a model that computes the penalty for deviating from the target state by taking the difference of the target state value(s) and the current value of the state [32]. Schemes 3 and 4 are based on the concept of dividing the state values according to ranges and assigning the highest score on the target range while assigning decreasing reward scores as the binning range moves away from the target range. Scheme 5 is built on top of a binary reward system; hence, it represents a binary scheme. The coding of these reward schemes was performed in the environment definition via Python scripting.

2.5.4. Hardware Setup

The computer specifications are the following. Unit: Dell Precision 7670 Mobile Workstation; CPU: Intel Core i7-12850HX, 25 MB cache, 24 threads, 16 core, 2.1 GHz–4.8 GHz; memory: 64 GB DDR5; GPU: NVIDIA GeForce RTX 3080Ti with 16 GB GDDR6. All computational work was implemented in this laptop computer running the AspenTech version 12 software suite containing the AspenPlus^® and Anaconda Navigator version 2.5 software [33], containing Python version 3.10 and all the necessary Python packages, such as PyTorch (CUDA-enabled), Gymnasium by OpenAI, and the pywin32 package [19] that interfaces with Windows to access the AspenPlus^® file. Note that even though this study used a GPU-enabled computer to accelerate PyTorch computations, a CPU-only implementation of PyTorch can still accomplish the computations, but at slower computation speeds. Nonetheless, the computations in this work do not require specialized computer hardware beyond a typical hardware requirement for running the AspenPlus^® software.

2.5.5. Code and Documentation of the Work Performed

The necessary codes used in this work have been organized and stored in an online repository via GitHub, as detailed in the Data Availability section. There are two main code files: (1) a Python class file that defines all functions used to read data from (output data) and write data to (input data) AspenPlus^® via Python, and (2) Jupyter Notebook (version 6.29.5) files that document the implementation of the SAC RL.

2.6. SAC RL Runs

Since RL training is very sensitive to randomization of the initial weight in the deep neural networks in the actor and in the critic components of the SAC model, a set of ten runs was implemented for each learning setting. Each run had a unique random number generator (RNG) index value used for all PyTorch computations, which consequently fixed the random number used in initializing the weights of the neural networks. The ten unique RNG indices can be seen in the accompanying Jupyter Notebook files in the project online repository (see Data Availability section). This deliberate randomization using known RNG indices accomplishes two crucial tasks: (1) testing the robustness in SACR RL models and (2) allowing for the repeatability of the randomized runs and results.

All runs were set to a maximum of 500 iterations per SAC learning run, where each iteration consisted of one cycle of the SAC model reading the current state and reward values of the environment (column model in AspenPlus^®), SAC model making a prediction for the next actions, and sending the predicted actions to the environment for simulation (to reach the next state and reward for the next cycle). Even though the traditional fields implementing RL (e.g., computer science, robotics, etc.) would usually set maximum iterations up to the range of millions [13], distillation column design iteration becomes impractical if the maximum iterations become very high, so our setting of a maximum of 500 iterations per SAC learning run was based on a practical basis. The data collected from each run as discussed above would not require extensive downstream processing, as these data by themselves are sufficient to evaluate the performance of SAC RL.

Furthermore, each run started with an untrained SAC RL model. This design of the study aimed to demonstrate that even an untrained SAC RL model can learn to optimize within a reasonable length of time (max of 500 iterations).

2.7. Data Collection and Analysis

There were numerous data that could be collected in this study, because the nature of the environment is highly data intensive, inherent in computer-based design of distillation columns in AspenPlus^®, and because the SAC RL training is data intensive. This prompted the research team to focus on collecting only key data that would enable evaluation of the performance of SAC RL in column internals design. The data for the following variables were saved and used for discussion: levels of the action variables, levels of the state variables, levels of the reward variables, and runtime per iteration. The saving of these datasets was coded in the run codes to eliminate user errors and streamline the analysis.

3. Results and Discussion

The results are presented in the following order: Section 3.1 covers the results to demonstrate the feasibility of using SAC RL for distillation column design; Section 3.2 shows the effect of the reward scheme on the performance of SAC RL; Section 3.3 discusses the effect of column diameter on the performance of SAC; Section 3.4 shows the effect of the SAC’s hyperparameters setting on its performance; and Section 3.5 covers the comparison of SAC RL with established optimization algorithms.

3.1. Feasibility of Using SAC RL for Distillation Column Internals Design

The first question that must be answered is, “Does SAC RL work as an automation method in optimizing distillation column internals design?” The answer is yes, it works, but there are pertinent trends in its performance that must be addressed, which we now cover as we proceed. The evidence for this claim can be seen in Figure 4 and Figure 5, which were the results when SAC RL was implemented on the column with a fixed uniform (in both TOP and BOT sections) diameter of 6 feet. Using the reward Scheme 1, the SAC model learned how to optimize the column design as the training progressed, converging to predicting column designs that maximize the reward (max of 200), as seen in Figure 4A, and selecting actions (column design parameter levels) that resulted in % flooding within the target range of 80–90% (Table 2, Scheme 1), as seen in Figure 4B,C. Initially, the SAC model did not have any optimal policy in determining the best next actions, because the deep neural networks in the critic and actor components were randomly initialized. This is evident in the very large standard deviation of the reward values and state variable values at the beginning of the training (Figure 4A–C). However, as the cycle of training–prediction–implementation steps progressed, the model learned the best actions that result in high rewards and the bad actions that result in low rewards, and it eventually converges to optimal policies (actor and critic) that favor the prediction of actions (Figure 5) that result in favorable $f_{\%}$ values resulting in the highest rewards (Figure 4). This trend of convergence is very evident in the last 100 iterations, i.e., 400–500 iteration steps accompanied by smaller (narrower) standard deviations of reward and state variable values (Figure 4).

Another crucial aspect of the design task is the duration of the SAC RL automated optimization, because chemical process design can be time sensitive [3]. The summary of runtime (in seconds) as shown in Figure 4D indicates that the average time per design iteration by the SAC model is 1 s (on the laptop computer used in this study—see Section 2.5.4, Hardware Setup, for details on the computer). This means that a single run with a maximum of 500 iterations (Figure 4) takes only around 10 min, which may be significantly lower than the time it takes to iterate the design manually.

With the numerous iterations that resulted in $f_{\%}$ values within the target range [80%, 90%) as shown in Figure 4B,C, the SAC-based optimization essentially exhaustively discovered numerous ways in which the column internals parameters can be set while achieving target design specification, i.e., $f_{\%}\in [80\mathrm{\%}, 90\mathrm{\%})$. The action space trends shown in Figure 5 show the corresponding column design parameters for these design iterations that meet the design specification.

3.2. Effect of Reward Scheme on the Performance of SAC

There is evidence in the literature of RL algorithms that show the impact of the reward scheme on the performance of RL models [29,30,31], and this was evaluated in the study. The results of testing the effects of the reward schemes (Table 2) are summarized in

Table 3. Summary of fractions of column design iterations that are below the cut-off values for % flooding values $f_{\%}$. The last 100 iteration steps of each of the 10 runs was used, i.e., 1000 samples per reward model. The following SAC RL model settings were used: $\tau =0.05$, $\gamma =0.99$, $\alpha =0.2$, and $\mathrm{replay} \mathrm{buffer} \mathrm{length}=100$. Column diameter was fixed at 6 feet.

Reward Model	Column Section	$\mathbf{Mean} {\mathit{f}}_{\mathit{\%}}$	$\mathbf{Std}. \mathbf{Dev}. {\mathit{f}}_{\mathit{\%}}$	$\mathbf{Fraction} \mathbf{of} {\mathit{f}}_{\mathit{\%}}<100$	$\mathbf{Fraction} \mathbf{of} {\mathit{f}}_{\mathit{\%}}<90$	$\mathbf{Fraction} \mathbf{of} {\mathit{f}}_{\mathit{\%}}<85$	$\mathbf{Fraction} \mathbf{of} {\mathit{f}}_{\mathit{\%}}<80$
Scheme 1	TOP	88.5	8.7	0.955	0.826	0.251	0
	BOT	88.8	8.6	0.945	0.796	0.216	0
Scheme 2	TOP	87.9	7.1	0.963	0.837	0.293	0
	BOT	88.2	6.1	0.974	0.835	0.146	0
Scheme 3	TOP	87.7	6.0	0.971	0.841	0.29	0
	BOT	88.0	5.6	0.976	0.823	0.214	0
Scheme 4	TOP	93.8	14.6	0.851	0.587	0.111	0
	BOT	94.0	14.6	0.845	0.57	0.102	0
Scheme 5	TOP	101.9	21.8	0.686	0.371	0.067	0
	BOT	102.4	22.0	0.686	0.359	0.052	0

. The data in Table 3 were calculated using the results of the last 100 iteration steps (iteration steps 400 to 500) in each of the 10 runs. The idea behind this data analysis is that the learning of the SAC RL model should be improving as it progresses in the sequence of iterations and that the best version of the model is achieved toward the end of the learning process. Expectation is based on the proven convergence of SAC RL as more training data are introduced to the model [13,14]. Therefore, computing the fraction of these last steps of iterations that meet the specific $f_{\%}$ cut-off would be warranted. Furthermore, the number of steps considered was 100 because the replay buffer length for the SAC model used in these runs is 100.

Looking at the mean and standard deviation results (Table 3), it can be observed that only Schemes 1, 2, and 3 achieved $f_{\%}$ values within the target interval [80%, 90%), while Scheme 4 and Scheme 5 produced $f_{\%}$ values above this target range. This supports prior observations in RL models that the reward scheme affects the performance of the RL model, even the SAC RL model. An interesting comparison is between Scheme 3 and Scheme 4, because these two schemes are similar in terms of the binning intervals of the $f_{\%}$ values but with Scheme 4 reward values 10 times lower than the Scheme 3 reward values. This means that the scaling of the reward values significantly affects the performance of the SAC RL model.

The consequence of the effects of the reward scheme is also evident in the fraction of the iterations that meet cut-off values of the design metric $f_{\%}$ (Table 3). The best-performing reward schemes, Schemes 1, 2, and 3, predicted column design specifications with $f_{\%}<90$ around at least 0.80 fraction of the iterations. The poor-performing schemes predicted column design specifications with $f_{\%}<90$ at low fractions: ~0.50 for Scheme 4 and ~0.35 for Scheme 5. Scheme 5, which represents a “binary scheme”, is the worst reward scheme.

3.3. Effect of Column Diameter on the Performance of SAC

With the positive results of implementing SAC RL, as shown in the previous section, it was imperative to evaluate the limitation of the approach, and a direct way of accomplishing this is by varying the diameter of the column. This is because column flooding is inversely proportional to column diameter, as shown in the theory of distillation column design [2,15,16], i.e., $f_{\%}~{\displaystyle \frac{1}{{\left(\mathrm{Column} \mathrm{Diameter}\right)}^2}}$. One set of experiments used fixed column diameter values of 5 ft, 6 ft, and 7 ft; another set included the column diameter ($A_{11}$) in the action space used by the SAC RL model. The results of implementing SAC RL in these experimental settings are shown in Figure 6.

It can be seen in Figure 6 that the SAC RL converges in terms of its reward values and the resulting state variable values as model training progresses across all column diameter settings. In general, the flooding levels $f_{\%}$ decrease as the column diameter increases. This is consistent with the theoretical relation that $f_{\%}$ is inversely proportional to column diameter (relation shown above). When the column diameter is lowest, at 5 feet, the converged SAC RL model is just approaching 100% flooding from above (Figure 6A,B), with actual values reaching only ~110% flooding as the lower limit. This means that it is impossible to have a 5 ft column that can operate at the target [80%, 90%) range of $f_{\%}$. This impossibility can be quickly checked by implementing the SAC RL, as was done here, instead of the user spending a very long time manually iterating for something that cannot happen (if column diameter is fixed at 5 ft).

At this point, it is imperative to ask the question, “If the flooding $f_{\%}$ is inversely proportional to column diameter, then why not just keep on increasing the diameter?” The answer comes from the limitations of fluid dynamics in the column when diameter is set too large. The issue of “weeping” can occur when there is a large pool of liquid on the tray (due to large column diameter) while vapor pressure is too low due to a larger active area of vapor–liquid contact resulting from a larger diameter [2,15]. This study does not cover solving the issue of weeping (it can be covered in a future extension of SAC RL in a similar approach). This then begs the question, “How was this trade-off in column diameter and flooding included in the SAC RL model?” This trade-off was included by implementing a specialized reward scheme by modifying reward Scheme 1 (Table 2) as follows (and let us call this Scheme 6 for quick referencing):

$$r={\displaystyle \frac{1}{3}}{r}_{TOP}+{\displaystyle \frac{1}{3}}{r}_{\mathrm{BOT}}+{\displaystyle \frac{1}{3}}{d}_{dia}$$

(2)

where $d_{dia}={\displaystyle \frac{100}{\max \left(\mathrm{Col}. \mathrm{Diamater}\right)-\min \left(\mathrm{Col}. \mathrm{Diamater}\right)}}\left[\min \left(\mathrm{Col}. \mathrm{Diameter}\right)-A_{11}\right]$, and $r_k$ is the reward in section $k$ of the column. For each $k$:

$$r_k=\left\{\begin{array}{cc}\hfill 100,& {80\le f}_{\%}<90\hfill \\ \hfill 100-2d_{err},& f_{\%}<80\hfill \\ \hfill 100-2d_{err},& f_{\%}\ge 90\hfill \end{array}\right.$$

(3)

where $d_{err}=\left|f_{\%}-85\right|$.

This reward Scheme 6 uses the maximum and minimum possible values of the column diameter, i.e., [4 ft, 9 ft] to compute a scaling factor adjusted to a magnitude of 100 as the maximum but assigned a negative value, i.e., $\left[\min \left(\mathrm{Col}. \mathrm{Diameter}\right)-A_{11}\right]\in [0, -\mathrm{\infty })$, to penalize the reward $r$ when the predicted action value for column diameter $A_{11}$ becomes too large. In essence, $d_{dia}\in \left[-100, 0\right]$ is a penalty value that becomes more negative as column diameter increases. Scheme 6 bounds reward $r$, resulting in the convergence of the SAC RL model, as shown in Figure 6G–I.

3.4. Effect of SAC Hyperparameters on Performance

Inherent in training RL models is the tuning of hyperparameters in order to further refine the models. This section covers the results of evaluating the effect of four hyperparameters: $\alpha$, $\gamma$, $\tau$, and replay buffer length. Of the numerous hyperparameters, these four have direct effects on SAC RL because of their roles in the fundamental equations of SAC.

Table 4. Summary of fractions of column design iterations that are below the cut-off value $f_{\%}=90$ for flooding values $f_{\%}$. The last 100 iteration steps of each of the 10 runs were used, i.e., 1000 samples per reward model. The runs implemented were for a column diameter = 6 feet and reward model Scheme 1.

SAC Setting	Hyperparameter				$\mathbf{Fraction} \mathbf{of} {\mathit{f}}_{\mathit{\%}}<90$	$\mathbf{Mean} {\mathit{f}}_{\mathit{\%}}$	$\mathbf{Std}. \mathbf{Dev}. {\mathit{f}}_{\mathit{\%}}$
	$\mathit{\alpha }$	$\mathit{\tau }$	$\mathit{\gamma }$	Replay Buffer Length
1	0.2	0.05	0.99	50	0.945	85.3	4.0
2	0.2	0.05	0.9	50	0.918	85.9	3.6
3	0.5	0.05	0.9	100	0.881	86.8	5.3
4	0.2	0.05	0.9	100	0.886	86.8	6.3
5	0.2	0.05	0.99	100	0.826	88.5	8.7
6	0.2	0.01	0.99	50	0.978	84.8	2.1
7	0.5	0.01	0.9	50	0.865	86.8	5.2
8	0.1	0.01	0.9	50	0.881	86.7	5.1
9	0.2	0.01	0.9	50	0.938	85.4	3.2
10	0.5	0.01	0.9	100	0.916	86.5	5.3
11	0.1	0.01	0.9	100	0.947	85.8	3.2
12	0.2	0.01	0.9	100	0.932	86.4	6.1
13	0.2	0.01	0.99	100	0.870	87.5	7.7

summarizes the flooding results for the top section (TOP) of the column set at a fixed diameter of 6 feet. Even though data were also collected for the bottom section (BOT) of the column, space limitations in the manuscript prompted reducing the presented data to half; hence, Table 4 shows data for the TOP section.

The effect of hyperparameter $\alpha$ can be seen by first comparing the results between Settings 10 and 11 and between Settings 7 and 8 (Table 4). The general trend is that a lower $\alpha$ value of 0.1 will produce higher fractions of runs that fall within the target condition $f_{\%}<90$ compared with when $\alpha$ is set higher, at 0.5. However, when comparing between Settings 8 and 9, it can be seen that $\alpha =0.1$ has a lower fraction $f_{\%}<90$ compared with $\alpha =0.2$. This means that there is an optimal value for $\alpha$ in the range 0.1 to 0.5, and this value may be close to 0.2. Since the value of $\alpha$ is a measure of the fraction of entropy contribution in the loss function (see Equation (1)), this observed trend means that the term in the loss function for maximizing entropy should not be zero, but it also should not be a very large part of the loss function. This is just a fitting observation because this entropy term is a unique feature of the SAC in comparison with other RL algorithms implementing actor-critic networks [13,23].

The effect of hyperparameter $\tau$ can be seen by first comparing the results between Settings 3 and 10 and between Settings 4 and 12 (Table 4). Both pairs show that a lower value of $\tau$ results in a higher fraction $f_{\%}<90$. If there is any optimum value somewhere, these data pairs cannot support such a possibility—perhaps a more extensive scan of the hyperparameter space can discover an optimum, but this is beyond the scope of this study. Since hyperparameter $\tau$ represents the smoothing coefficient for the target Q-functions, setting its value to zero may not be a good idea, because this number controls the update of the Q-function deep neural network weights.

The effect of hyperparameter $\gamma$ can be seen by first comparing the results between Settings 1 and 2, between Settings 4 and 5, and between Settings 12 and 13 (Table 4). It can be seen in these pairs that the trends for fraction $f_{\%}<90$ do not follow a clear, consistent trend. It is possible that there are confounding effects in the SAC RL training that cannot be easily isolated to be the sole effect of $\gamma$. Therefore, there is no conclusive trend for the effect of this hyperparameter based on the data.

The effect of the hyperparameter ‘replay buffer length’ can be seen by first comparing results between Settings 2 and 4, between Settings 6 and 13, and between Settings 8 and 11 (Table 4). It can be seen in these pairs that the trends for fraction $f_{\%}<90$ do not follow a clear, consistent trend. Similar to hyperparameter $\gamma$, it is possible that there are confounding effects in the SAC RL training that cannot be easily isolated to be the sole effect of ‘replay buffer length’. Therefore, there is no conclusive trend for the effect of this hyperparameter based on the data.

3.5. Comparison of SAC RL Algorithm with Established Optimization Algorithms

A pertinent question at this point is, “How does SAC RL performance compare with established optimization algorithms?” This question is necessary to answer, because this study is proposing to use SAC RL in an optimization task that has been heavily studied [34]. The tested algorithms are the following: Nelder–Mead [35], Broyden–Fletcher–Goldfarb–Shanno (BFGS) [36], Sequential Least Squares Programming (SLSQP) [37], dual annealing [38], and simplicial homology global optimization (SHGO) [39], which were implemented using their pre-built functions in the Python module SciPy [40]. To simplify the comparative analysis, the target flooding of $f_{\%}=85$ was used, which was based on reward Scheme 1 used for the SAC RL. Each algorithm was implemented in ten runtimes, with each run assigned a unique RNG index. Figure 7 summarizes the results from experiments grouped into (1) at a fixed column diameter of 6 ft, as shown in Figure 7A−F; and (2) at variable column diameters with a penalty on the diameter the same as that of Scheme 6, as shown in Figure 7G,H.

It can be seen that each of the algorithms tested exhibited unique performance in terms of optimizing to 85% flooding. Poor performance can be seen for BFGS and SLSQP, which exhibited early stops (Figure 7C,D), while dual annealing and SHGO completed the runs but with erratic behavior in various stages of the iterations (Figure 7E,F). The Nelder–Mead exhibited a smooth convergence to the target when the column diameter was fixed (Figure 7B). However, Nelder–Mead failed to converge to the target when the column diameter was included in the varied parameters (Figure 7H). We also want to note that dual annealing and SHGO were also tested when the column diameter was varied, but these algorithms exhibited erratic performance and failed to converge to the target. The SAC-RL algorithm was able to converge to the target in both the fixed column diameter (Figure 7A) and variable column diameter (Figure 7G) design scenarios. These trends show the advantage of SAC RL over established optimization algorithms for the task of optimizing distillation column internals design. Due to SAC RL’s maximum entropy term, it continues to explore as it learns, resulting in the prevention of exploitation of actions that may be sub-optimal.

4. Conclusions

This study focused on developing an automated method using soft actor-critic (SAC) reinforcement leaning for optimizing the design of distillation columns, and this aim was achieved, as supported by the results. The integration of SAC RL into AspenPlus^® model optimization was achieved by using Python codes written to allow a cyclic interaction between the distillation column model AspenPlus^® and the SAC model in PyTorch. The specific chemical system used as a case study was a binary mixture of ethanol and water, and the RadFrac module in AspenPlus^® was used to enable rigorous column internals design. The results clearly support the following findings: (1) SAC RL works as an automation approach for the design of distillation column internals, (2) the reward scheme in the SAC model significantly affects SAC performance, (3) column diameter is a significant constraint in achieving column internals design specifications in flooding, and (4) SAC hyperparameters have varying effects on SAC performance.

This study also demonstrated that an untrained SAC RL model can quickly learn how to optimize the design of a distillation column. This means that the algorithm can be implemented as a one-shot RL for column internals design. This has significant implications for possible future implementation of the technique as an integral part of computer-based chemical process modeling, e.g., if this technique becomes part of AspenPlus^® software (and other similar computer packages). That is, there is no need to install a pre-trained SAC RL model into a computer-based chemical process simulation package that would utilize this technique.