An Adaptive Linear Programming Algorithm with Parameter Learning

Abstract

When dealing with engineering design problems, designers often encounter nonlinear and nonconvex features, multiple objectives, coupled decision making, and various levels of fidelity of sub-systems. To realize the design with limited computational resources, problems with the features above need to be linearized and then solved using solution algorithms for linear programming. The adaptive linear programming (ALP) algorithm is an extension of the Sequential Linear Programming algorithm where a nonlinear compromise decision support problem (cDSP) is iteratively linearized, and the resulting linear programming problem is solved with satisficing solutions returned. The reduced move coefficient (RMC) is used to define how far away from the boundary the next linearization is to be performed, and currently, it is determined based on a heuristic. The choice of RMC significantly affects the efficacy of the linearization process and, hence, the rapidity of finding the solution. In this paper, we propose a rule-based parameter-learning procedure to vary the RMC at each iteration, thereby significantly increasing the speed of determining the ultimate solution. To demonstrate the efficacy of the ALP algorithm with parameter learning (ALPPL), we use an industry-inspired problem, namely, the integrated design of a hot-rolling process chain for the production of a steel rod. Using the proposed ALPPL, we can incorporate domain expertise to identify the most relevant criteria to evaluate the performance of the linearization algorithm, quantify the criteria as evaluation indices, and tune the RMC to return the solutions that fall into the most desired range of each evaluation index. Compared with the old ALP algorithm using the golden section search to update the RMC, the ALPPL improves the algorithm by identifying the RMC values with better linearization performance without adding computational complexity. The insensitive region of the RMC is better explored using the ALPPL—the ALP only explores the insensitive region twice, whereas the ALPPL explores four times throughout the iterations. With ALPPL, we have a more comprehensive definition of linearization performance—given multiple design scenarios, using evaluation indices (EIs) including the statistics of deviations, the numbers of binding (active) constraints and bounds, the numbers of accumulated linear constraints, and the number of iterations. The desired range of evaluation indices (DEI) is also learned during the iterations. The RMC value that brings the most EIs into the DEI is returned as the best RMC, which ensures a balance between the accuracy of the linearization and the robustness of the solutions. For our test problem, the hot-rolling process chain, the ALP returns the best RMC in twelve iterations considering only the deviation as the linearization performance index, whereas the ALPPL returns the best RMC in fourteen iterations considering multiple EIs. The complexity of both the ALP and the ALPPL is O(n²). The parameter-learning steps can be customized to improve the parameter determination of other algorithms.

1. Introduction

Solution algorithms for solving engineering design problems fall into two categories: (i) an optimizing strategy or (ii) a satisficing strategy. For typical engineering design problems with nonlinear, nonconvex properties that cannot be solved in a straightforward way using simplex, Lagrange multipliers, or other gradient-based solution algorithms, when applying the optimizing strategy, designers need to solve problems approximately, for example, using pattern search methods [1], particle swarm optimization [2], memetic algorithms [3], penalty function methods [4], or metaheuristic algorithms [5]. When methods in Category (i) are used, the solutions are desired to be near-Pareto solutions, often on or close to the boundary of the feasible space (bounded by constraints) but not necessarily at a vertex of the feasible space. Whereas when methods in Category (ii) are used, such as sequential linear programming [6] or FEAST [7], the solutions are always at one or multiple vertices of the linearized (approximated) solution space. Since the solution always includes a vertex, we are able to use information of the dual problem to explore the solution space [6]. Typically, in engineering design problems the number of constraints in the primal formulation exceeds the number of variables [8]. In these cases, solving the dual formulation is quicker [6]. For optimization strategies, there are state-of-the-art surveys summarizing and classifying linearization techniques thoroughly, such as in [9]. Linearization techniques are classified into two groups—nonlinear equations replaced by an exact equivalent LP formulation or linear approximations which find the equivalent of a nonlinear function with the least deviation around the point of interest or separate straight-line segments.

In this paper, our focus is on a method in Category (ii), the satisficing strategy, namely, the compromise decision support problem (cDSP) and adaptive linear programming (ALP) algorithm [6], because it is more appropriate for engineering design problems by giving satisficing solutions and exploring solution space. Satisficing solutions are near-optimal and on the vertices of the approximated linear problems. The reasons are summarized in Section 1.1 and Section 1.2.

1.1. Frame of Reference

There are special requirements for engineering design methods. Engineering design is a task that occupies multiple designers with different fields of expertise to make decisions that meet a variety of requirements [10]. Designers may define objective functions using different methods—utility theory [11], game theory [12], analytic hierarchy process (AHP) [13], Pareto-optimal methods [14], etc. For optimization problems with one or multiple objective function(s) to be minimized or maximized, one can define such problems based on maximization of expected utility. Designers accept assumptions and simplifications when using utility theory to manage a problem, sometimes without realizing it. Such assumptions include the abstraction of mathematical relationships between variables to be 100% accurate and not evolve, the levels of fidelity of different sub-models or segments of a model are the same, the design preferences among multiple objectives are fully captured, etc. [15]. For many problems, these assumptions are wrong and this may cause problems. For example, when implementing an optimal solution, a system may not give optimal output, because of the inaccurate equations used in the model, or because uncertainty breaks the equilibrium of any Karush–Kuhn–Tucker (KKT) conditions, thus destroying the optimality of the solution [16].

In addition, when the problem is nonconvex, specifically, when any nonzero linear combination of the constraints is more convex than the objective function [17], Category (i) methods cannot return a feasible solution because the sufficient KKT condition cannot be satisfied [15]. In this case, formulating the problem using the compromise decision support problem (cDSP) and solving it using the adaptive linear programming (ALP) algorithm is a way to identify satisficing solutions that meet the necessary KKT condition [15].

1.2. Mechanisms to Ensure That the cDSP and ALP Find Satisficing Solutions

There are five mechanisms in the cDSP and ALP that enable designers to identify satisficing solutions to nonlinear, nonconvex engineering design problems, especially when optimization methods fail. We summarize the mechanisms in

Table 1. The mechanisms in the cDSP and ALP to find satisficing solutions [16].

Mechanisms	Advantage	Assumption Removed
Using goals and minimizing deviation variables instead of objectives	At a solution point, only the necessary KKT condition is met, whereas the sufficient KKT condition does not have to be met. Therefore, designers have a greater chance of finding a solution and a lower chance of losing a solution due to parameterizable and/or unparameterizable uncertainties.	Assumption 1
Using second-order sequential linearization	Designers can have a balance between linearization accuracy and computational complexity.	Assumption 2
Using accumulated linearization	Designers can manage nonconvex problems and deal with highly convex, nonlinear problems relatively more accurately.	Assumption 2
Combining interior-point search and vertex search	Designers can avoid getting trapped in local optima to some extent and identify satisficing solutions which are relatively insensitive when the starting points change.	Assumption 1
Allowing some violations of soft requirements, such as the bounds of deviation variables	Designers can manage rigid requirements and soft requirements in different ways to ensure feasibility. As a result, goals and constraints with different scales can be managed.	Assumptions 1 and 2

. Two assumptions in standard optimization methods make optimal solutions unavailable for some nonlinear, nonconvex problems:

Assumption 1.

Mathematical models are 100% complete and accurate abstractions of physical models, so the optimal solution to the mathematical problem is optimal for the physical problem.

Assumption 2.

The convexity degree of at least one nonzero linear combination of all constraints is higher than the convexity degree of the objective function.

However, for Assumption 1, designers need boundary information to deal with uncertainties, but such information is normally unavailable [18]. Using the cDSP and ALP, designers can identify satisficing solutions by removing the two assumptions.

There are differences between the satisficing construct using cDSP-ALP and optimization including its variant goal programming. Solutions are usually obtained by identifying the Pareto frontier consisting of nondominated or near-optimal solutions using optimization solution algorithms. The formulation of design problems using a satisficing strategy, namely, the compromise decision support problem (cDSP), has the key features that allow designers to identify satisficing solutions that meet the necessary KKT condition but not the sufficient condition.

The format of a nonlinear optimization problem is this: for a given objective function $\mathrm{f}\left(\mathrm{x}\right)$, Euler and Lagrange developed the Euler–Lagrange equation forming the second-order ordinary differential equations ${\nabla }_{\mathrm{xx}}^2\mathrm{f}\left(\mathrm{x}\right)$ to facilitate finding the stationary solutions. The value of the variables that maximize $\mathrm{f}\left(\mathrm{x}\right)$ within the feasible set $\mathcal{F}$ is the solution to the optimization problem, where $\mathcal{F}$ is the set bounded by constraints and bounds. The format of an optimization problem $\mathbb{O}$ can be represented as follows. $\mathrm{x}$ is the vector of decision variables as real numbers. ${\mathrm{g}}_{\mathrm{i}}\left(\mathrm{x}\right)$ is the i^th inequality constraint. ${\mathrm{h}}_{\mathrm{j}}\left(\mathrm{x}\right)$ is the i^th equality constraint. Any point $\mathrm{x}$ that is a local extremum of the set mapped by multiplying active equations with a nonnegative vector is a local optimum of $\mathbb{O}$ [19], denoted as ${\mathrm{x}}^{*}$. The elements of such a nonnegative vector are Lagrange multipliers, μ and λ.

The format of an optimization problem $\mathbb{O}$:

Given

$$\begin{array}{l}\mathrm{f}: {\mathbb{R}}^{\mathrm{n}}\to \mathbb{R},\mathcal{F}\subseteq {\mathbb{R}}^{\mathrm{n}}\\ \hspace{1em}\hspace{1em}\mathcal{F}=\left\{\mathrm{x}\in {\mathbb{R}}^{\mathrm{n}}|{\mathrm{g}}_{\mathrm{i}}\left(\mathrm{x}\right)\ge 0, \mathrm{i}=1,\dots ,\mathrm{m},{\mathrm{h}}_{\mathrm{j}}\left(\mathrm{x}\right)=0, \mathrm{j}=1,\dots ,\ell \right\}\end{array}$$

find

$$\begin{array}{l}{\mathrm{x}}^{*}: \mathrm{f}\left({\mathrm{x}}^{*}\right)\succcurlyeq \mathrm{f}\left(\mathrm{x}\right), \forall \mathrm{x}\in \mathcal{F}\end{array}$$

One variant of optimization is goal programming. The format of a goal programming problem ${\mathbb{O}}^{\mathrm{goal}}$ is represented as follows. A target value $\mathrm{T}$ is predefined for the objective function $\mathrm{f}\left(\mathrm{x}\right)$ as the right-hand side value, so the objective becomes an equation, and we call it a goal. ${\mathrm{d}}^{-}$ and ${\mathrm{d}}^{+}$ are deviation variables measuring the underachievement and overachievement of the goal towards its target. The problem is solved by minimizing the deviation variables, which is minimizing the difference between $\mathrm{f}\left(\mathrm{x}\right)$ and $\mathrm{T}$. In other words, goal programming is aimed at finding $\mathrm{T}$’s closest projection on $\mathcal{F}$.

The format of a goal programming problem ${\mathbb{O}}^{\mathrm{goal}}$:

Given

$$\begin{array}{l}\mathrm{f}: {\mathbb{R}}^{\mathrm{n}}\to \mathbb{R},\mathcal{F}\subseteq {\mathbb{R}}^{\mathrm{n}}\\ \hspace{1em}\hspace{1em}\mathcal{F}=\left\{\mathrm{x}\in {\mathbb{R}}^{\mathrm{n}}|{\mathrm{g}}_{\mathrm{i}}\left(\mathrm{x}\right)\ge 0, \mathrm{i}=1,\dots ,\mathrm{m},{\mathrm{h}}_{\mathrm{j}}\left(\mathrm{x}\right)=0, \mathrm{j}=1,\dots ,\ell ,{\mathrm{d}}^{-}·{\mathrm{d}}^{+}=0, 0\le {\mathrm{d}}^{\mp }\le 1,\mathrm{Goal}: \mathrm{f}\left(\mathrm{x}\right)+{\mathrm{d}}^{-}-{\mathrm{d}}^{+}=\mathrm{T}\right\}\end{array}$$

find

$${\mathrm{x}}^{*}:{\mathbb{P}}_{\mathrm{x}\in \mathcal{F}}\left(\mathrm{Goal}: \mathrm{f}\left(\mathrm{x}\right)=\mathrm{T}\right)$$

In the cDSP, elements of mathematical programming and goal programming are combined. A cDSP $\mathbb{C}$ is represented as follows. For a nonlinear cDSP, we first linearize the nonlinear equations, including nonlinear constraints and nonlinear goal. Therefore, the nonlinear cDSP first becomes a linear problem with a linear goal ${\mathrm{Goal}}^{\mathrm{linear}}$, a linear feasible space ${\mathcal{F}}^{\mathrm{li}}$ bounded by linear constraints $\mathrm{g}{\left(\mathrm{x}\right)}^{\mathrm{li}}\ge 0$ and $\mathrm{h}{\left(\mathrm{x}\right)}^{\mathrm{li}}=0$. Thus, using a cDSP, we seek the closest projection from the linear goal set onto a linear feasible set. We define the solution as a satisficing solution, and we use ${\mathrm{x}}^{\mathrm{s}}$ to denote it.

The format of a cDSP $\mathbb{C}$:

Given

$$\begin{array}{l}\mathrm{f}: {\mathbb{R}}^{\mathrm{n}}\to \mathbb{R},\mathcal{F}\subseteq {\mathbb{R}}^{\mathrm{n}}\\ {\mathcal{F}}^{\mathrm{li}}=\left\{\mathrm{x}\in {\mathbb{R}}^{\mathrm{n}}|{\mathrm{g}}_{\mathrm{i}}{\left(\mathrm{x}\right)}^{\mathrm{li}}\ge 0, \mathrm{i}=1,\dots ,\mathrm{m},{\mathrm{h}}_{\mathrm{j}}{\left(\mathrm{x}\right)}^{\mathrm{li}}=0, \mathrm{j}=1,\dots ,\ell ,{\mathrm{d}}^{-}·{\mathrm{d}}^{+}=0, 0\le {\mathrm{d}}^{\mp }\le 1,{\mathrm{Goal}}^{\mathrm{li}}:\frac{\mathrm{f}{\left(\mathrm{x}\right)}^{\mathrm{li}}}{\mathrm{T}}+{\mathrm{d}}^{-}-{\mathrm{d}}^{+}=1\right\}\end{array}$$

find

$${\mathrm{x}}^{\mathrm{s}}:{\mathbb{P}}_{\mathrm{x}\in {\mathcal{F}}^{\mathrm{li}}}\left({\mathrm{Goal}}^{\mathrm{li}}:\mathrm{f}{\left(\mathrm{x}\right)}^{\mathrm{li}}=\mathrm{T}\right)$$

The difference between ${\mathrm{x}}^{*}$ and ${\mathrm{x}}^{\mathrm{s}}$ is that ${\mathrm{x}}^{*}$ conforms to both the necessary (first-order) and sufficient (second-order) KKT conditions, whereas ${\mathrm{x}}^{\mathrm{s}}$ conforms to the necessary KKT condition but may not conform to the sufficient KKT condition. This is because the second derivative of the linear equations, ${\mathrm{Goal}}^{\mathrm{li}}$, ${\mathrm{g}}_{\mathrm{i}}{\left(\mathrm{x}\right)}^{\mathrm{li}}\ge 0$, and ${\mathrm{h}}_{\mathrm{j}}{\left(\mathrm{x}\right)}^{\mathrm{li}}=0$, degenerates—as a result, no uncertainty may affect the feasibility of ${\mathrm{x}}^{\mathrm{s}}$ because there is no uncertainty to break the equilibrium of the second-order Lagrange equation. In addition, when the convexity of $\mathcal{F}$ is greater than the convexity of the $\mathrm{f}\left(\mathrm{x}\right)$, ${\mathrm{x}}^{*}$ may not be identified as the second-order Lagrange equation has no solution, but ${\mathrm{x}}^{\mathrm{s}}$ is obtainable because the second-order KKT condition is irrelevant. The relationship among optimal solutions, satisficing solutions, and near-optimal solutions is illustrated in Figure 1.

In Chapter 2 of [15], the author gives five examples to demonstrate how the mechanisms work (we have a demonstration of the comparison of five examples in this video: https://www.youtube.com/watch?v=7apDZO-9A74, accessed on 21 November 2020, and a tutorial of using DSIDES to formulate and solve a problem in this video: https://www.youtube.com/watch?v=tUpVC97Y1L8, accessed on 25 November 2020. In this paper, since our focus is to fill a gap in the ALP, we give only one example in Section 1.3 to show one of the advantages versus the optimization method. Information such as applications using cDSP and ALP to obtain satisficing solutions can be found in [20].

1.3. An Example and Explanation Using KKT Conditions

We use a simple example with multiple goals (objectives), nonlinear and nonconvex equations, and the goal targets with various degrees of achievability. We formulate the problem using optimization and the cDSP in

Table 2. The optimization model and compromise DSP of the example.

Optimizing

Satisficing

Objective Functions $${\mathrm{f}}_1\left(\mathrm{x}\right)=\cos ({{\mathrm{x}}_1}^2+{{\mathrm{x}}_2}^3)$$ $${\mathrm{f}}_2\left(\mathrm{x}\right)=25·{\left({\mathrm{x}}_1-2\right)}^3+50·{\left({\mathrm{x}}_2-2\right)}^3+50·{\mathrm{x}}_1·{{\mathrm{x}}_2}^2$$ Constraints and Bounds $$\mathrm{s}.\mathrm{t}. \left\{\begin{array}{c}\begin{array}{c}{\mathrm{x}}_1·{\mathrm{x}}_2\le 1\\ {\mathrm{f}}_1\left(\mathrm{x}\right)\ge 0\\ {\mathrm{f}}_2\left(\mathrm{x}\right)\ge 0\\ 0\le {\mathrm{x}}_1\le 2\\ 0\le {\mathrm{x}}_2\le 2\end{array}\end{array}\right.$$ Combination of Objective Functions $$\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{Max} {\sum }_{\mathrm{i}=1}^2{\mathrm{w}}_{\mathrm{i}}·{\mathrm{f}}_{\mathrm{i}}\left(\mathrm{x}\right)$$

Given $$\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\mathrm{x}}_1,{ \mathrm{x}}_2, {{\mathrm{d}}_1}^{\pm }, {{\mathrm{d}}_2}^{\pm }$$ $$\hspace{1em}\hspace{1em}\hspace{1em}{\mathrm{f}}_1\left(\mathrm{x}\right)=\cos \left({{\mathrm{x}}_1}^2+{{\mathrm{x}}_2}^3\right)$$ $${\mathrm{f}}_2\left(\mathrm{x}\right)=25·{\left({\mathrm{x}}_1-2\right)}^3+50·{\left({\mathrm{x}}_2-2\right)}^3+50·{\mathrm{x}}_1·{{\mathrm{x}}_2}^2$$ Find $$\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\mathrm{x}}_1,{ \mathrm{x}}_2, {{\mathrm{d}}_1}^{\mp },{{\mathrm{d}}_2}^{\mp }$$ Satisfy Goals: $$\frac{{\mathrm{f}}_1\left(\mathrm{x}\right)}{1.2}+{{\mathrm{d}}_1}^{-}-{{\mathrm{d}}_1}^{\mathrm{+}}=1$$ $$\frac{{\mathrm{f}}_2\left(\mathrm{x}\right)}{400}+{{\mathrm{d}}_2}^{-}-{{\mathrm{d}}_2}^{\mathrm{+}}=1$$ Constraints: $$\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\mathrm{x}}_1·{\mathrm{x}}_2\le 1$$ $$\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\mathrm{f}}_1\left(\mathrm{x}\right)\ge 0$$ $$\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\mathrm{f}}_2\left(\mathrm{x}\right)\ge 0$$ $$\hspace{1em}\hspace{1em}\hspace{1em}{{\mathrm{d}}_{\mathrm{i}}}^{-}·{{\mathrm{d}}_{\mathrm{i}}}^{\mathrm{+}}=0,\mathrm{i}=1, 2$$ Bounds: $$\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}0\le {\mathrm{x}}_1,{ \mathrm{x}}_2\le 2$$ $$\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}0\le {{\mathrm{d}}_1}^{\pm }, {{\mathrm{d}}_2}^{\pm }\le 1$$ Minimize $$\mathrm{Merit} \mathrm{function} \mathrm{Z}={\sum }_{\mathrm{i}=1}^2{\mathrm{w}}_{\mathrm{i}}·\left({{\mathrm{d}}_{\mathrm{i}}}^{-}+{{\mathrm{d}}_{\mathrm{i}}}^{+}\right)$$

.

The methods used in the optimizing and satisficing strategies are listed in

Table 3. Methods for comparison of the two strategies.

	Strategy	Optimizing	Satisficing
Item
Model formulation construct		Mathematical programming Goal programming	Compromise decision support problem
Solution algorithm		Constrained optimization by linear approximation (COBYLA) algorithm	Adaptive linear programing (ALP) algorithm
		Trust-region constrained (trust-constr) algorithm
		Sequential least squares programming (SLSQP) algorithm
		Nondominated sorting generation algorithm II/III (NSGA II)
Solver		Python SciPy.optimize	DSIDES [6]

. For optimizing methods, we used the “SciPy.optimize” package (https://docs.scipy.org/doc/scipy/reference/optimize.html, accessed on 1 January 2008). There are ten algorithms in the package. We use three of them to solve the example problem because they are the only relevant ones. We do not use the Nelder–Mead, Powell (Powell’s conjugate direction method) [21], or conjugate gradient (CG) methods [22] or the Broyden–Fletcher–Coldfarb–Shannon (BFGS) algorithm [23] because they cannot easily manage problems with constraints. The Newton conjugate gradient (Newton-CG) method [24,25], L-BFGS-B (an extension BFGS for large-scale, bounded problems) [26], and TNC (truncated Newton method or Hessian-free optimization) [26] either cannot deal with problems without Jacobians (when using Newton-CG, even setting the Jacobian as false, and the algorithm may not work without a provided Jacobian partially because the default temporary memory of Jacobian cannot be cleared; see: https://stackoverflow.com/questions/33926357/jacobian-is-required-for-newton-cg-method-when-doing-a-approximation-to-a-jaco, accessed on 15 May 2019) or return infeasible solutions without recognizing that they are infeasible. Therefore, in SciPy we use the constrained optimization by linear approximation (COBYLA) algorithm [27,28], Trust-constr (https://docs.scipy.org/doc/scipy/reference/optimize.minimize-trustconstr.html, accessed on 1 January 2008), and sequential least squares programming (SLSQP) (https://docs.scipy.org/doc/scipy/reference/optimize.minimize-slsqp.html, accessed on 1 January 2008) to solve the example problem.

We select nondominated sorting genetic algorithm II (NSGA II) [29] as a verification method to compare and evaluate the performance of our selected optimization methods and the satisficing method in Table 3. We choose NSGA II as the verification method because it can solve problems with the complexities of the example problem in Table 2—nonlinearity, nonconvexity, multiple objectives or goals, and various achievability of the goals. We use NSGA II in MATLAB. Some readers may wonder, since NSGA II can solve the complexities often incorporated in engineering design problems, why do we study a satisficing algorithm, the cDSP, and ALP to manage engineering design problems? The reason is that we observe that NSGA II has the following drawbacks that may prevent designers from acquiring insight to improve the design formulation and exploring the solutions space:

First, NSGA II cannot give designers information to improve the model, such as the bottlenecks in the model, the sensitivity of each segment of the model, the rationality of the dimensions of the model, etc.

Second, the performance of NSGA II, including convergence speed, optimality of solutions, and diversity of solutions, is sensitive to hyperparameter settings. Hyperparameters, such as the population size and generation number, must be predefined. However, usually designers only assume that a larger population size or a larger number of generations returns better solutions, but they may not know how large is “good enough”. Designers need to tune those hyperparameters in NSGA II, but it requires much higher computational power than the cDSP and ALP do.

Therefore, we choose NSGA II in order to assess the optimality and diversity of the solutions returned by the tested methods (in Table 3), but we still recommend designers use the satisficing method to manage engineering design problems.

Through applying the three chosen optimizing methods, the satisficing method, and the verification method, NSGA II, we obtain the results and summarize them in

Table 4. Solutions to the example problem—dominated solutions for each weight are in italics; for each method, the best solution of each design scenario is marked using a capital letter (A’, B’, C’, D’, E’, F’, G’, H’, and J’) [15].

Weight	Starting Point	COBYLA		Trust-Constr		SLSQP		ALP		NSGA II/III—Population (P) = 20/50
		Solution	${\mathbf{\sum }}_{\mathit{i}\mathbf{=}\mathbf{1}}^{\mathbf{2}}$wi · fi(x)	Solution	${\mathbf{\sum }}_{\mathit{i}\mathbf{=}\mathbf{1}}^{\mathbf{2}}$wi · fi(x)	Solution	${\mathbf{\sum }}_{\mathit{i}\mathbf{=}\mathbf{1}}^{\mathbf{2}}$wi · fi(x)	Solution	${\mathbf{\sum }}_{\mathit{i}\mathbf{=}\mathbf{1}}^{\mathbf{2}}$wi · fi(x)	Solution	${\mathbf{\sum }}_{\mathit{i}\mathbf{=}\mathbf{1}}^{\mathbf{2}}$wi · fi(x)
(1, 0)	(0.5, 1)	Cannot manage nonconvex equations with bounds		All solutions violate one or more constraints		All solutions violate one or more constraints		A’ (0.51,1.82)	1	P = 20: G’ (0.55, 1.85) P = 50: A’ (0.51, 1.82)	P = 20: 0.99 P = 50: 1
	(0, 0)
	(2, 0.5)
(0, 1)	(0.5, 1)							B’ (0.51, 1.96)	15.27	P = 20: H’ (0.52, 1.92) P = 50: B’ (0.51, 1.96)	P = 20: 14.86 P = 50: 15.36
	(0, 0)
	(2, 0.5)
(0.5, 0.5)	(0.5, 1)							C’ (0.55, 1.82)	7.5	P = 20: H’ (0.52, 1.92) P = 50: D’ (0.53, 1.87)	P = 20: 7.69 P = 50: 7.78
	(0, 0)
	(2, 0.5)
(0.7, 0.3)	(0.5, 1)							C’ (0.55, 1.82)	4.9	P = 20: C’ (0.55, 1.82) P = 50: E’ (0.53, 1.88)	P = 20: 4.9 P = 50: 5.01
	(0, 0)
	(2, 0.5)
(0.3, 0.7)	(0.5, 1)							C’ (0.55, 1.82)	10.01	P = 20: J’ (0.54, 1.85) P = 50: F’ (0.53, 1.89)	P = 20: 10.49 P = 50: 10.6
	(0, 0)
	(2, 0.5)

. Since there are two objectives (goals), we use different weights to combine them linearly, so we show the results for each weight. We also use different starting points for the solution searching to show whether a solution algorithm is sensitive to the starting point. We visualize the solutions in objective space and x-f(x) space in Figure 2 and Figure 3. The three optimization algorithms cannot return any feasible solutions. Due to the difference in the scale of the two objectives, one cannot use optimizing algorithms to solve the example problem by linearly combining them, because (i) the objective with a large scale dominates the other objective(s) and (ii) a linearized function of the weighted sum objective in a local area can be singular.

For a multi-objective (multi-goal) problem with nonconvex functions, when the scale of the objectives varies largely, optimizing algorithms cannot return feasible solutions, whereas a satisficing strategy may allow designers to identify adequate solutions. This is explained using the KKT conditions hereafter.

When using optimizing algorithms to solve optimization problems, the first-order derivative of the Lagrange equation with respect to decision variable x, which is a function of the parameters $\mathcal{P}$ of the model (the coefficients in objectives and constraints), decision variables x (if any objective or constraint is nonlinear), Lagrange multipliers μ and λ, and weights $𝓅$, combining the multiple goals, is shown in Equation (1).

$${\nabla }_{\mathrm{x}}\mathrm{L}\left(\mathrm{x},\mathsf{\mu },\mathsf{\lambda }\right)=\mathcal{y}\left(\mathcal{P},\mathrm{x},\mathsf{\mu },\mathsf{\lambda }, 𝓅\right)$$

(1)

For a satisficing strategy, the first-order derivative of the Lagrange equation contains only the coefficients of the deviation variables in the objective, since only deviation variables d^∓ constitute the objective (not the decision variables, x). For a $\mathsf{\kappa }$-goal cDSP with m inequality constraints g(x) and $\ell$ equality constraints h(x), if we use weights to combine the $\mathsf{\kappa }$ goals G(x,d), i.e., using the Archimedean strategy to manage a multi-goal cDSP, then the coefficients in the first-order Lagrange equation are only the weights and $\tau$—$\tau$ is the Lagrange multiplier of the goal functions; see Equation (2).

$$\begin{array}{l}{\nabla }_{\mathrm{d}}\mathrm{L}\left({\mathrm{x}}^{\mathrm{s}},\mathrm{d},\mathsf{\mu },\mathsf{\lambda },\tau \right)={\nabla }_{\mathrm{d}}\mathcal{z}\left(\mathrm{d}\right)+{{\displaystyle \sum }}_{\mathrm{i}=1}^{\mathrm{m}}{\mathsf{\mu }}_{\mathrm{i}}{\nabla }_{\mathrm{d}}{\mathrm{g}}_{\mathrm{i}}\left({\mathrm{x}}^{\mathrm{s}}\right)-{{\displaystyle \sum }}_{\mathrm{j}=1}^{\ell }{\mathsf{\lambda }}_{\mathrm{j}}{\nabla }_{\mathrm{d}}{\mathrm{h}}_{\mathrm{j}}\left({\mathrm{x}}^{\mathrm{s}}\right)-{{\displaystyle \sum }}_{\mathrm{k}=1}^{\mathsf{\kappa }}{\tau }_{\mathrm{k}}{\nabla }_{\mathrm{d}}\mathrm{G}\left({\mathrm{x}}^{\mathrm{s}},\mathrm{d}\right)\\ =\mathcal{y}\left(𝓅,\tau \right)\end{array}$$

(2)

when using optimization, the second-order Lagrange equation may still have parameters and decision variables due to nonlinearity; see Equation (3). For satisficing, the second-order Lagrange equation with respect to deviation variables degenerates to zero because the objective of a cDSP is a linear combination of deviation variables; see Equation (4). That is why satisficing solutions do not need to meet the second-order KKT conditions.

$${\nabla }_{\mathrm{xx}}^2\mathrm{L}\left(\mathrm{x},\mathsf{\mu },\mathsf{\lambda }\right)={\nabla }_{\mathrm{x}}\mathcal{y}\left(\mathcal{P},\mathrm{x},\mathsf{\mu },\mathsf{\lambda },𝓅\right)={\mathcal{y}}^{\prime }\left({\mathcal{P}}^{\prime },\mathrm{x}\right)$$

(3)

$${\nabla }_{\mathrm{dd}}^2\mathrm{L}\left({\mathrm{x}}^{\mathrm{s}},\mathrm{d},\mathsf{\mu },\mathsf{\lambda },\tau \right)={\nabla }_{\mathrm{d}}\left(\mathcal{y}\left(𝓅,\tau \right)\right)\equiv 0$$

(4)

Although both optimal solutions ${\mathrm{x}}^{*}$ and satisficing solutions ${\mathrm{x}}^{\mathrm{s}}$ meet the first-order KKT conditions, the chance of maintaining the first-order KKT conditions for the two strategies under uncertainties varies. If any uncertainty with probability P takes place to an item ℑ in the first-order equation that destroys its equilibrium, we denote it as $\mathrm{Pr}\left(\tilde{\Im }|\mathrm{P}\right)$. For an N-dimension, Q-parameter (here, we define a coefficient or an intercept of a constraint or an objective as a parameter. A parameter has a given value (either a constant value or a stochastic value) and the value does not depend on any decision variables), and $\mathsf{\kappa }$-goal problem, using optimizing strategy, the source of $\tilde{\Im }$ can be decision variables $\widetilde{{\mathrm{x}}_{\mathrm{n}}}$, Lagrange multipliers $\widetilde{{\mathsf{\mu }}_{\mathrm{i}}}$ and $\widetilde{{\mathsf{\lambda }}_{\mathrm{j}}}$, and weights $\widetilde{{𝓅}_{\mathrm{k}}}$; for satisficing strategy, the source of $\tilde{\Im }$ can only be the weights $\widetilde{{𝓅}_{\mathrm{k}}}$ and the Lagrange multipliers for the goals $\widetilde{{\tau }_{\mathrm{k}}}$. If and only if none of the items under the uncertainty breaks the equilibrium of the first-order equation, then the optimal/satisficing solution is still optimal/satisficing under this uncertainty. For an N-dimension, Q-parameter, and $\mathsf{\kappa }$-goal problem, the probabilities of maintaining an optimal solution and a satisficing solution under Uncertainty $\mathrm{P}$ are given in Equations (5) and (6), respectively.

$$\mathrm{Pr}\left({\mathrm{x}}^{*}|\mathrm{P}\right)\approx {{\displaystyle \prod }}_{\mathrm{q}=1}^{\mathrm{Q}}\left[1-\mathrm{Pr}\left(\widetilde{{\mathcal{P}}_{\mathrm{q}}}|\mathrm{P}\right)\right]{{\displaystyle \prod }}_{\mathrm{n}=1}^{\mathrm{N}}\left[1-\mathrm{Pr}\left(\widetilde{{\mathrm{x}}_{\mathrm{n}}}|\mathrm{P}\right)\right]{{\displaystyle \prod }}_{\mathrm{i}=1}^{\mathrm{m}}\left[1-\mathrm{Pr}(\widetilde{{\mathsf{\mu }}_{\mathrm{i}}}|\mathrm{P})\right]{{\displaystyle \prod }}_{\mathrm{j}=1}^{\ell }\left[1- \mathrm{Pr}(\widetilde{{\mathsf{\lambda }}_{\mathrm{j}}}|\mathrm{P})\right]{{\displaystyle \prod }}_{\mathrm{k}=1}^{\mathsf{\kappa }}\left[1-\mathrm{Pr}(\widetilde{{𝓅}_{\mathrm{k}}}|\mathrm{P})\right]$$

(5)

$$\mathrm{Pr}({\mathrm{x}}^{\mathrm{s}}|\mathrm{P})\approx {{\displaystyle \prod }}_{\mathrm{k}=1}^{\mathsf{\kappa }}\left[1-\mathrm{Pr}(\widetilde{{𝓅}_{\mathrm{k}}}|\mathrm{P})\right]{{\displaystyle \prod }}_{\mathrm{k}=1}^{\mathsf{\kappa }}\left[1-\mathrm{Pr}(\widetilde{{\tau }_{\mathrm{k}}}|\mathrm{P})\right]$$

(6)

As the value of any probability is in the range of [0, 1], the more items on the right-hand side we multiply (the more items the probability depends on), the lower the probability becomes. The items in Equation (6) are fewer than those of Equation (5). Hence, the chance of maintaining an optimal solution under Uncertainty $\mathrm{P}$ is often smaller than the chance of maintaining a satisficing solution with the same uncertainty; Equation (7).

$$\mathrm{Pr}\left({\mathrm{x}}^{*}|\mathrm{P}\right)\le \mathrm{Pr}\left({\mathrm{x}}^{\mathrm{s}}|\mathrm{P}\right)$$

(7)

In summary, using the satisficing strategy, we are less likely to lose a solution due to uncertainty for nonconvex problems with multiple objectives that have various scales; see Equations (8) and (9). Using satisficing, designers can deal with nonconvex, multi-objective problems that may be incomplete or inaccurate and with uncertainties, which helps remove Assumptions 1 and 2.

$${\left[\mathrm{Pr}\left(\mathcal{S}\ne \varnothing |\left\{\begin{array}{c}\mathrm{Nonconvexity}\\ \mathrm{objectives} \mathrm{with} \mathrm{various} \mathrm{scales}\end{array}\right\}\right)\right]}_{\mathrm{Satisficing}}\ge$$

$${\left[\mathrm{Pr}\left(\mathcal{S}\ne \varnothing |\left\{\begin{array}{c}\mathrm{Nonconvexity}\\ \mathrm{objectives} \mathrm{with} \mathrm{various} \mathrm{scales}\end{array}\right\}\right)\right]}_{\mathrm{Optimizing}}$$

(8)

$${\left[\mathrm{Pr}\left({\mathrm{x}}^{\mathrm{s}}|\mathrm{P}\right)\right]}_{\mathrm{Satisficing}}\ge {\left[\mathrm{Pr}\left({\mathrm{x}}^{*}|\mathrm{P}\right)\right]}_{\mathrm{Optimizing}}$$

(9)

1.4. A Limitation in the ALP to Be Improved

Just like NSGA II, the ALP also has a parameter, the reduced move coefficient, or RMC, that may impact the solution space, especially for multi-goal problems. The problems as simple as the example problem in Section 1.3 is not sensitive to RMC setting, but more complicated problems are. We use a test problem of steel rod manufacturing to illustrate it in Section 4.

The RMC is a fractional step size [6] defining the starting point for the next iteration. In the ALP, the RMC has been set at 0.5 as a default value. In fact, in the ALP, a designer has no knowledge of the connection between the RMC and solution quality or the opportunity for improving solution efficiency by controlling the RMC. Improving the RMC determination is discussed in [30]. In this paper, we give more explanations about the motivation, the advantages of the satisficing strategy using the cDSP and ALP, and the benefits of parameter learning.

We introduce the ALP and discuss the limitation of using a fixed RMC in Section 2. In Section 3, we introduce parameter learning to make the RMC adaptive for each iteration. In Section 4, we use a test problem that is sensitive to the RMC value—the steel hot rod-rolling process—to demonstrate efficacy of the augmented ALP, that is, the adaptive linear programming algorithm with parameter learning (ALPPL). In Section 5, we summarize the contributions and comment on the generalization of parameter learning for use in gradient-based optimization methods.

2. How Does the ALP Work?

2.1. The Adaptive Linear Programming (ALP) Algorithm

The ALP algorithm is implemented in DSIDES. DSIDES is used to formulate and solve engineering design problems, and it is especially efficient in dealing with nonlinear problems [6]. In DSIDES, the nonlinear problem is formulated as a compromise decision support problem (cDSP). Then, the ALP is invoked to solve the nonlinear problem. The nonlinear problem is linearized in a synthesis cycle. The resulting linear problem is solved using the revised dual simplex algorithm. The synthesis cycle is repeated until the solution satisfies a set of stopping criteria or is terminated after a fixed number of iterations.

The ALP incorporates a local approximation algorithm [6,31], in which a secant plane of the paraboloid (with the second-order derivatives at the starting point as the coefficients) replaces the original nonlinear function. In Figure 4, we show two dimensions of a problem being approximated in two iterations (synthesis cycle). The weighted sum of the goals is ${\sum }_{\mathrm{k}\in \mathrm{K}}{\mathrm{W}}_{\mathrm{k}}·{\mathrm{G}}_{\mathrm{k}}$. The starting point ${\mathrm{X}}_0^0$ may not be in the feasible region. A random search or a Hooke–Jeeves pattern search can be invoked to identify a point ${\mathrm{X}}_1^0$ in the feasible area. In the i^th first iteration, the problem is linearized at ${\mathrm{X}}_1^0$.

In iteration i, Figure 4a, a projection of a nonlinear constraint NF_j onto a two-dimensional plane, X₁-X₂, is approximated at ${\mathrm{X}}_{\mathrm{i}}^0$, so an approximated constraint LF_i,j is obtained. Doing this for all nonlinear functions and framing a linear model, the revised simplex dual algorithm is used to obtain solution ${\mathrm{X}}_{\mathrm{i}}^{*}$. Using the RMC heuristic, we find the starting point of the next iteration ${\mathrm{X}}_{\mathrm{i}+1}^0$. In iteration i + 1, Figure 4b, the approximated linear constraints of both iterations LF_i,j and LF_i+1,j are accumulated, and a solution ${\mathrm{X}}_{\mathrm{i}+1}^{*}$ is returned and the starting point of iteration i + 2 is again defined using the RMC, ${\mathrm{X}}_{\mathrm{i}+1}^0$ and ${\mathrm{X}}_{\mathrm{i}+1}^{*}$.

In Figure 5a, we illustrate the two-step linear approximation method. First, NF_j (Paraboloid ABC) is approximated to NF_i,j″ (Paraboloid AB^*C^*) with the diagonal terms of its Hessian matrix at ${\mathrm{X}}_{\mathrm{i}}^0$ as coefficients. Then, NF_j″ is approximated to a secant Plane LF_i,j (Plane AB^*C^*). NF_j″ and LF_i,j are computed as follows.

NF_i,j″ is obtained using the second-order full derivatives at ${\mathrm{X}}_{\mathrm{i}}^0$, Equation (10), because the second-order partial derivatives have limited impact on the gradient.

$${{\mathrm{NF}}_{\mathrm{i},\mathrm{j}}}^{″}={\mathrm{NF}}_{\mathrm{j}}\left({\mathrm{X}}_{\mathrm{i}}^0\right)+{{\displaystyle \sum }}_{\mathrm{p}=1}^{\mathrm{n}}\left({\mathrm{x}}_{\mathrm{ip}}-{\mathrm{x}}_{\mathrm{ip}}^0\right){\left(\frac{\partial {\mathrm{NF}}_{\mathrm{j}}}{\partial {\mathrm{x}}_{\mathrm{ip}}}\right)}_0+\frac{1}{2}{{\displaystyle \sum }}_{\mathrm{p}=1}^{\mathrm{n}}{\left({\mathrm{x}}_{\mathrm{ip}}-{\mathrm{x}}_{\mathrm{ip}}^0\right)}^2{\left(\frac{{\partial }^2{\mathrm{NF}}_{\mathrm{j}}}{\partial {{\mathrm{x}}_{\mathrm{ip}}}^2}\right)}_0$$

(10)

From Equation (10), for the p^th dimension, the quadratic to be solved to obtain $\left({\mathrm{x}}_{\mathrm{ip}}-{\mathrm{x}}_{\mathrm{ip}}^0\right)$ is:

$${\mathrm{NF}}_{\mathrm{j}}\left({\mathrm{X}}_{\mathrm{i}}^0\right)+\left({\mathrm{x}}_{\mathrm{ip}}-{\mathrm{x}}_{\mathrm{ip}}^0\right){\left(\frac{\partial {\mathrm{NF}}_{\mathrm{j}}}{\partial {\mathrm{x}}_{\mathrm{ip}}}\right)}_0+\frac{1}{2}{\left({\mathrm{x}}_{\mathrm{ip}}-{\mathrm{x}}_{\mathrm{ip}}^0\right)}^2{\left(\frac{{\partial }^2{\mathrm{NF}}_{\mathrm{j}}}{\partial {{\mathrm{x}}_{\mathrm{ip}}}^2}\right)}_0=0$$

(11)

If Equation (11) has real roots, Figure 5a, by solving Equation (11) and selecting the root between Equations (12) and (13) with the smaller absolute value for each dimension, we obtain the intersection that is closer to the paraboloid in each dimension, such as B* and C*.

$${\left(\frac{\partial {\mathrm{NF}}_{\mathrm{j}}}{\partial {\mathrm{x}}_{\mathrm{ip}}}\right)}_0^{*}=\frac{-{\mathrm{NF}}_{\mathrm{j}}\left({\mathrm{X}}_{\mathrm{i}}^0\right){\left(\frac{{\partial }^2{\mathrm{NF}}_{\mathrm{j}}}{\partial {{\mathrm{x}}_{\mathrm{ip}}}^2}\right)}_0}{-{\left(\frac{\partial {\mathrm{NF}}_{\mathrm{j}}}{\partial {\mathrm{x}}_{\mathrm{ip}}}\right)}_0-\sqrt{{\left(\frac{\partial {\mathrm{NF}}_{\mathrm{j}}}{\partial {\mathrm{x}}_{\mathrm{ip}}}\right)}_0^2-2{\mathrm{NF}}_{\mathrm{j}}\left({\mathrm{X}}_{\mathrm{i}}^0\right){\left(\frac{{\partial }^2{\mathrm{NF}}_{\mathrm{j}}}{\partial {{\mathrm{x}}_{\mathrm{ip}}}^2}\right)}_0}}$$

(12)

$${\left(\frac{\partial {\mathrm{NF}}_{\mathrm{j}}}{\partial {\mathrm{x}}_{\mathrm{ip}}}\right)}_0^{*}=\frac{-{\mathrm{NF}}_{\mathrm{j}}\left({\mathrm{X}}_{\mathrm{i}}^0\right){\left(\frac{{\partial }^2{\mathrm{NF}}_{\mathrm{j}}}{\partial {{\mathrm{x}}_{\mathrm{ip}}}^2}\right)}_0}{-{\left(\frac{\partial {\mathrm{NF}}_{\mathrm{j}}}{\partial {\mathrm{x}}_{\mathrm{ip}}}\right)}_0+\sqrt{{\left(\frac{\partial {\mathrm{NF}}_{\mathrm{j}}}{\partial {\mathrm{x}}_{\mathrm{ip}}}\right)}_0^2-2{\mathrm{NF}}_{\mathrm{j}}\left({\mathrm{X}}_{\mathrm{i}}^0\right){\left(\frac{{\partial }^2{\mathrm{NF}}_{\mathrm{j}}}{\partial {{\mathrm{x}}_{\mathrm{ip}}}^2}\right)}_0}}$$

(13)

If Equation (11) has no real roots, Figure 5b, NF_i does not intersect Plane x, then the first-order derivative at ${\mathrm{X}}_{\mathrm{i}}^0$ is used, Equation (14).

$${\left(\frac{\partial {\mathrm{NF}}_{\mathrm{j}}}{\partial {\mathrm{x}}_{\mathrm{ip}}}\right)}_0^{*}={\left(\frac{\partial {\mathrm{NF}}_{\mathrm{j}}}{\partial {\mathrm{x}}_{\mathrm{ip}}}\right)}_0$$

(14)

Based on the intersections in each dimension, such as B* and C*, we obtain LF_i.

$${\mathrm{LF}}_{\mathrm{i},\mathrm{j}}={{\displaystyle \sum }}_{\mathrm{p}=1}^{\mathrm{n}}{\mathrm{x}}_{\mathrm{ip}}{\left(\frac{\partial {\mathrm{NF}}_{\mathrm{j}}}{\partial {\mathrm{x}}_{\mathrm{ip}}}\right)}_0^{*}-\left({{\displaystyle \sum }}_{\mathrm{p}=1}^{\mathrm{n}}{\mathrm{x}}_{\mathrm{ip}}^0{\left(\frac{\partial {\mathrm{NF}}_{\mathrm{j}}}{\partial {\mathrm{x}}_{\mathrm{ip}}}\right)}_0^{*}-{\mathrm{NF}}_{\mathrm{j}}\left({\mathrm{X}}_{\mathrm{i}}^0\right)\right)$$

(15)

Algorithm 1 summarizes the constraint accumulation algorithm. If the degree of convexity of NF_j is positive or slightly negative (greater than −0.015) at the starting point of the i^th iteration, and if the constraint is active in the (i − 1)^th iteration, that is, ${\mathrm{X}}_{\mathrm{i}-1}^{*}$ is on the surface of ${\mathrm{NF}}_{\mathrm{j}}$, then the accumulated constraints replace ${\mathrm{NF}}_{\mathrm{j}}$, Equation (16); otherwise, the single linear constraint in the i^th iteration is ${\mathrm{NF}}_{\mathrm{j}}$.

Algorithm 1. Constraint Accumulation Algorithm

In the ith iteration, for every j in J     if   $\frac{1}{\mathrm{n}}{\sum }_{\mathrm{p}=1}^{\mathrm{n}}\frac{{\partial }^2{\mathrm{NF}}_{\mathrm{j}}}{\partial {{\mathrm{x}}_{\mathrm{ip}}}^2}\le -0.015$       and ${\mathrm{NF}}_{\mathrm{j}}\left({\mathrm{X}}_{\mathrm{i}-1}^{*}\right)=0$

LF_i,j, = LF_i-1,j ∪ LF_i,j

(16)

Then, the revised simplex dual algorithm is invoked to solve the linear problem ${\mathrm{P}}_{\mathrm{i}}^{\mathrm{L}}$, so a solution ${\mathrm{X}}_{\mathrm{i}}^{*}$ is obtained. A point ${\mathrm{X}}_{\mathrm{i}+1}^0$, between the starting point ${\mathrm{X}}_{\mathrm{i}}^0$ and the solution ${\mathrm{X}}_{\mathrm{i}}^{*}$, becomes the starting point of the next iteration. The RMC is used to determine ${\mathrm{X}}_{\mathrm{i}+1}^0$, Equation (17).

$${\mathrm{X}}_{\mathrm{i}+1}^0={ \mathrm{X}}_{\mathrm{i}}^0+ \mathrm{RMC} ·\left({\mathrm{X}}_{\mathrm{i}}^{*}-{ \mathrm{X}}_{\mathrm{i}}^0\right)$$

(17)

2.2. The Reduced Move Coefficient

The RMC is in the range [0, 1] and it defaults to 0.5 based on experimental observations [6], but 0.5 may not be the best for every problem. An optional golden section search algorithm is added to progressively narrow the range of the RMC by cutting off the sub-range with large deviations or larger violations of the constraints until the range is too small to reduce, Figure 6. The golden section search algorithm is given as follows – visualized in Figure 6 and summarized a Algorithm 2.

Algorithm 2. Golden section search for updating the RMC

#Define Performance function using RMC to linearize the model and obtain the merit function value FUNCTION Performance (Model, RMC) Linearize Model using ALP with RMC into Linear_Model Solve Linear_Model using Dual Simplex RETURN Z #Define Golden Section Search function FUNCTION GoldenSectionSearchForRMC (Rmin, Rmax, Th): RMCa = Rmin + (0.382)*(Rmax − Rmin) RMCb = Rmin + (1 − 0.382)*(Rmax − Rmin) WHILE (RMCb − RMCa > Th):   #Compare the performance of using RMCa versus RMCb IF Performance (Original_Model, RMCa) < Performance (Original_Model, RMCb): RMC = RMCa Rmax = RMCb ELSE: RMC = RMCb Rmin = RMCa RETURN RMC #Initialize parameters Rmin = 0 Rmax = 1 Th = 0.0001 #Call the Golden Section Search Function GoldenSectionSearchForRMC (Rmin, Rmax, Th)

With the golden section search approach, the desired sub-range of the RMC may be missed if the performance oscillates in the range of the RMC. The criterion to evaluate the approximation performance is oversimplified—the deviations and the constraint violations. Given these limitations, in this paper, we propose to use parameter learning to determine the relation between the RMC value and solution quality with more evaluation criteria. Machine learning techniques are used to improve algorithms [32,33], so we leverage this idea to improve the ALP.

We hypothesize that by incorporating parameter learning in the ALP, we can update the RMC based on richer performance criteria. There are three steps to verify this hypothesis. In Section 3, we propose ALPPL to realize the three steps.

• Step 1. Identifying the criteria—to evaluate approximation performance.
• Step 2. Developing evaluation indices (EIs)—to quantify the approximation performance with RMC values.
• Step 3. Learning the desired range of each EI (DEI)—to tune the RMC.

In the next section, we discuss how parameter learning is used to dynamically change the value of the RMC.

3. Parameter Learning to Dynamically Change the RMC

In this section, the adaptive linear programming algorithm with parameter learning (ALPPL) is proposed, Figure 7. We add parameter updating, results extraction, performance evaluation, and feedback on parameter updating to the ALP.

3.1. Step 1—Identifying the Criteria

In ALPPL, we add more criteria to the two criteria considered in the golden section search—deviation and constraint violation. We use a list, EIs[best], to record all the evaluation indices.

• Criterion 1—deviation.

The deviation in every iteration is recorded and ranked among all iterations. The best deviation in all iterations is stored in EIs[best]. Through implementing multiple design scenarios (lexicographic and/or weight scenarios), we obtain the deviations for all scenarios and iterations, and the desired range DEI is updated accordingly.

• Criterion 2—robustness of solutions.

We extend the notion of “feasibility” to “robustness”, which means the solutions should be feasible under multiple design scenarios and relatively insensitive to model errors and uncertainties. We assume errors and uncertainties result in changes in the boundary of the feasible region. The nonlinear surface is a part of the boundary. Therefore, the robustness of the solution can be measured by how far it is away from the nonlinear surface. In Figure 8, we show two situations. The arrows indicate the direction of the feasible space bounded by the inequality constraints while minimizing the deviation. In Figure 8a, the solution ${\mathrm{x}}_{\mathrm{i}}^{*}$ is not on the boundary, so it is relatively insensitive to errors and variations of the model, and the nonlinear constraint NF_j is not an active constraint. In contrast, in Figure 8b, the solution is on the boundary and NF_j is an active constraint. The RMC affects the approximation in the next iteration, so we adjust the RMC to obtain more robust solutions.

• Criterion 3—approximation accuracy.

More accumulated constraints may not necessarily lead to a more accurate approximation. We want sufficient and useful accumulated constraints—Figure 9b—rather than many unnecessary accumulated constraints—Figure 9a.

In

Table 5. Criteria for the evaluation of approximation performance.

Criteria	Meaning and Representation
Weight-sum deviations	The unfulfilled percentage of the goals compared with their targets
Robustness	Whether a solution is away from the boundary of the system as defined by the nonlinear model
Approximation accuracy	Whether the nonlinear constraints are approximated well by a set of linear constraints in the sub-region that contains the solutions that fulfill the goals to an extent

, we summarize the criteria for approximation performance evaluation. Based on these criteria, we develop evaluation indices (EIs) in Section 3.2

3.2. Step 2—Developing the Evaluation Indices (EIs)

To manage different design preferences for multiple goals, we obtain solutions using multiple design scenarios. As the design scenarios are discrete and cannot enumerate all situations, we use the limited, discrete solutions to predict a satisficing solution space.

By implementing multiple design scenarios, we acquire the results of the weight-sum deviations, active constraints, accumulated constraints, etc., using which, we develop statistical-based evaluation indices (EIs) as shown in

Table 6. Develop evaluation indices (EIs)—mean (μ) and standard deviation (SD) (σ).

Criteria	Information	EIs	Meaning
Weight-sum deviation	$\mathrm{Z}={{\displaystyle \sum }}_{\mathrm{k}\in \mathrm{K}}{\mathrm{W}}_{\mathrm{k}}·\left({\mathrm{d}}_{\mathrm{k}}^{-}+{\mathrm{d}}_{\mathrm{k}}^{+}\right)$	μz	The average weight-sum deviations
		σz	The stability of the weight-sum deviations when changing design scenarios
Robustness	The number of active bounds:	μNab	The average sensitivity of the solutions under all design scenarios to variable bounds
		σNab	The stability of the sensitivity of the solutions to variable bounds when changing design scenarios
	The number of active original constraints:	μNaoc	The average sensitivity of the solutions under all design scenarios to original constraints
		σNaoc	The stability of the sensitivity of the solutions to original constraints when changing design scenarios
Approximation accuracy	The number of accumulated constraints:	μNacc	The average complexity of the approximated problem under all design scenarios
		σNacc	The stability of the complexity of the approximated problem when changing design scenarios
	The number of iterations:	μNit	The average convergence speed under all design scenarios
		σNi	The stability of the convergence speed when changing design scenarios

. Statistical-based quality control has been used to monitor model profiles [34], so we leverage the statistics to obtain EIs.

Our development of the EIs is based on the index for the robust concept exploration method, error margin index (EMI) [31], and design capability index (DCI) [35]. This is based on the central limit theorem that the results as samplings of each criterion follow Gaussian distributions, and the sampling statistics represent their characteristics. Therefore, we use the mean (μ) and the standard deviation (σ) as EIs and tune the RMC by minimizing the μ and σ of each EI.

Index for evaluating the weight-sum deviation—μ_Z and σ_Z. To maintain the goal fulfillment (one minus weight-sum deviation) relatively insensitive to design scenario changes, we control the center and spread of the sample solutions by minimizing μ_Z and σ_Z.

Index for evaluating the robustness—μ_Nab, σ_Nab, μ_Naoc, and σ_Naoc. Using the simplex algorithm, we obtain the vertex solution to the approximated linear problem. At a vertex solution, there is at least one active constraint (AOC) or active bound (AB) of the approximated linear problem, but we prefer fewer AOCs and ABs so the chance of losing a solution due to potential errors or variations is relatively small, so we minimize the μ_Nab, σ_Nab, μ_Naoc, and σ_Naoc.

Index for evaluating the computational complexity—μ_Nacc, σ_Nacc, μ_Nit, and σ_Nit. As the approximation accuracy is not necessarily improved by increasing the number of accumulated constraints (Nacc) or the number of approximation iterations (Nit), we need to find the ranges of Nacc and Nit associated with good weight-sum deviations and robustness. We desire Nacc and Nit to be acceptable under all design scenarios and be insensitive to scenarios changing, so we measure μ_Nacc, σ_Nacc, μ_Nit, and σ_Nit and identify their appropriate ranges.

In summary, we tune the RMC by satisfying the EIs—μ_Nab, σ_Nab, m_Naoc, σ_Naoc, μ_Nacc, σ_Nacc, μ_Nit, and σ_Nit— to obtain desired ranges (DEI). Then, the RMC tuning becomes: “given” the parameters and variables, “find” the value of decision variables that can “satisfy” constraints and bounds and “minimize” an objective. Here, we rank the EIs in the i^th iteration among all iterations and choose the first κ items with minimum values. These are the most important κ where EIs attain their best and most stable performance. Algorithm 3 is a summary of the RMC parameter learning algorithm.

Algorithm 3. #Define Parameter-learning function

FUNCTION ParameterLearning(Model, Design scenarios, RMC, EIs, κ, I): WHILE i < I:  IF RMC[i-1] = RMC_best:     RMC[i] = average(RMC[i-2], RMC_best)  ELSE:     RMC[i] = average(RMC[i-1], RMC_best)  FOR N Design Scenarios:     Linearize Model using ALP with RMC[i] into Linear_Model     Solve Linear_Model using Dual Simplex  RETURN EIs[i]  FOR j in Range(0, i):     Rank EIs[i][j] from minimum to maximum     DEI = EIs[i][round(κ*i)]  IF (number of EIs[i] ∈ DEI for all Design scenarios) > (number of EIs[Cycle of RMC_best] ∈ DEI for all Design scenarios):     RMC_best = RMC[i] RETURN RMC_best #Initialize parameters Model = Original_Model Design scenarios = N weight scenarios RMC [0] = 0.5 EIs = [μZ, σZ, μNab, σNab, μNaoc, σNaoc, μNacc, σNacc, μNit, σNit] Z, Nab, Naoc, Nacc, Nit = 0 DEI = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0] κ = 0.6 I = 50 #Call Parameter-Learning function FUNCTION ParameterLearning(Model, Design scenarios, RMC, EIs, κ, I)

3.3. Step 3—Learning the DEI to Tune the RMC

We identify the desired range of the EIs (DEI), learn the connections between the RMC and the EIs, and bring the EIs into the DEI by setting the RMC. To make the process efficient, we combine off-line learning and the on-line learning. First, we use off-line learning using a sample of RMC values to initialize the DEI and parameters and then adopt on-line learning to update the DEI and tune the RMC. In Figure 10, we illustrate the two processes (Rectangles B and C) and show their relationship with the synthesis cycle A.

During parameter initialization (B), we generate sample values of RMC (B1). By running the synthesis cycle (A) with each RMC value, we obtain the corresponding EIs (B2) and evaluate them (B3). We choose the RMC value associated with the best EIs as the starting RMC value for tuning (B3-1) and remove the EIs that are insensitive to RMC changes (B3-2). We initialize the DEI based on the sample results (B3-3) to allow a certain proportion (e.g., 75%) of RMC values to fall into the DEI. We choose the best RMC among the sample (B3-4) to start the on-line learning. These results (B3-1 to B3-4) are aggregated as the “actions to be taken” (B3-5) and the input of the RMC tuning cycle (C).

For the RMC tuning cycle (C), with an RMC value (C1-1), we run the synthesis cycle (A) and obtain the results (C2). By evaluating the results using the DEI and comparing with previous iterations (C3), we determine the next RMC value (C3-1), update the DEI if necessary—either restrict or relax the DEI based on the tradeoffs among EIs (C3-2)—and update the best RMC if necessary (C3-3). These evaluation results are aggregated (C3-4) for judging whether to stop iterating (C4)—the program either goes to the next iteration of RMC tuning with the aggregated results (C3-4) as input or stops with the best RMC as the returned value. The stopping criteria include the number of total iterations and the number of iterations without updating the best RMC. This part is summarized as Algorithm 4.

Algorithm 4. The RMC parameter-learning algorithm

1 Given: the best RMC sample value, updating rules 2 Initialize: t = 0, best = the best RMC sample value, RMC0 = the best RMC sample value, the maximum iteration number T, stopping criterion 2 = {best has not been updated in n iterations} 3 While t = do T // Define stopping criterion 1 4   RMCt = Next_RMC 5    Run synthesis cycle

4. A Test Problem

In this section, we apply ALPPL to an industry-inspired problem, the integrated design of a hot-rolling process chain for the production of a steel rod [36]. It is a nonlinear problem, and the RMC value has a significant impact on the result.

4.1. The Hot Rod-Rolling Process Chain

Hot rod rolling is a multi-stage manufacturing process in which a reheated billet, slab, or bloom produced after the casting process is further thermo-mechanically processed by passing it through a series of rollers [36]. During the thermo-mechanical processing, there is an evolution of microstructure of the material, in this case, steel. Columnar grains in the cast material are broken down to equiaxed grains. Along with the evolution of grain size, there is a phase transformation of the steel. The phase transformation is predominant during the cooling stage that follows the hot rod-rolling process chain. The transformation of the austenite phase of steel to other phases like ferrite, pearlite, or martensite takes place during this stage. The final microstructure of the material after the rolling and cooling process defines the mechanical properties of the product.

Many plant trials are required to produce a new steel grade with improved properties and performance. These trials are usually expensive and time-consuming. Hence, there is a need to address the problem from a simulation-based design perspective to explore solutions reaching multiple conflicting property/performance goals. The requirement is to produce steel rods with improved mechanical properties such as yield strength (YS), tensile strength (TS), and hardness (HV). These mechanical properties are defined by the microstructure after cooling, which includes the phase fractions (ferrite and pearlite phases are only considered in this problem), pearlite interlamellar spacing, ferrite grain size, and chemical compositions of steel. Nellippallil et al. [36] identify the microstructural requirements after the cooling stage to meet the mechanical properties of the rod. The microstructural requirements are to achieve a high ferrite fraction value, low pearlite interlamellar spacing, and low ferrite grain size values within the defined ranges. The requirement is to carry out the integrated design of the material and the process by managing the cooling rate (cooling process variable), final austenite grain size after rolling (rolling microstructure variable), and the chemical compositions of the material. Hence, we explore the solution space of the defined variables using ALPPL to meet the target values identified for the microstructure after the cooling stage such that the mechanical property requirements of the steel rod are met. Our focus in this paper is to use this example in improving the solution algorithm rather than the design of the material and the manufacturing. The initial design formulation of the problem is shown as follows.

Given

Target values for microstructure after cooling

Ferrite grain size target, D_α,Target = 8 μm

Ferrite fraction target, X_f,Target = 0

Pearlite interlamellar spacing target, S_0,Target = 0.15

Find

System variables

X₁ cooling rate (CR)

X₂ austenite grain size (D)

X₃ the carbon concentration ([C])

X₄ the manganese concentration after rolling ([Mn])

Deviation variables

${\mathrm{d}}_{\mathrm{i}}^{-},{ \mathrm{d}}_{\mathrm{i}}^{+}$, i =1,2,3

Satisfy

System constraints

Minimum ferrite grain size constraint          D_α ≥ 8 μm

(18)

Maximum ferrite grain size constraint          D_α ≤ 20 μm

(19)

Minimum pearlite interlamellar spacing constraint      S_o ≥ 0.15 μm

(20)

Maximum pearlite interlamellar spacing constraint      S_o ≤ 0.25 μm

(21)

Minimum ferrite phase fraction constraint (manage banding)      X_f ≥ 0.5

(22)

Maximum ferrite phase fraction constraint (manage banding)  X_f ≤ 0.9

(23)

Maximum carbon equivalent constraint    C_eq = (C + Mn)/6;  C_eq ≤ 0.35

(24)

Mechanical Property Constraints

Minimum yield strength constraint        YS ≥ 250 MPa

(25)

Maximum yield strength constraint        YS ≤ 330 MPa

(26)

Minimum tensile strength constraint        TS ≤ 480 MPa

(27)

Maximum tensile strength constraint        TS ≥ 625 MPa

(28)

Minimum hardness constraint        HV ≥ 130

(29)

Maximum hardness constraint        HV ≤ 150

(30)

System goals

The target values for system goals are identified in [36] and are listed under the keyword Given above.

$$\mathrm{Goal} 1: \mathrm{Achieve} \mathrm{ferrite} \mathrm{grain} \mathrm{size} \mathrm{target} \hspace{1em}\hspace{1em}\hspace{1em}{{\mathrm{D}}_{\mathsf{\alpha }}}_{\mathrm{Target}}/{\mathrm{D}}_{\mathsf{\alpha }}\left({\mathrm{X}}_{\mathrm{i}}\right)+{\mathrm{d}}_1^{+}-{\mathrm{d}}_1^{-}=1$$

(31)

$$\mathrm{Goal} 2: \mathrm{Achieve} \mathrm{ferrite} \mathrm{fraction} \mathrm{target} \hspace{1em}\hspace{1em}\hspace{1em}{\mathrm{X}}_{\mathrm{f}}\left({\mathrm{X}}_{\mathrm{i}}\right)/{{\mathrm{X}}_{\mathrm{f}}}_{\mathrm{Target}}+{\mathrm{d}}_2^{-}-{\mathrm{d}}_2^{+}=1$$

(32)

$$\mathrm{Goal} 3: \mathrm{Achieve} \mathrm{pearlite} \mathrm{interlamellar} \mathrm{spacing} \mathrm{target} \hspace{1em}\hspace{1em}{{\mathrm{S}}_{\mathrm{o}}}_{\mathrm{Target}}/{\mathrm{S}}_{\mathrm{o}}\left({\mathrm{X}}_{\mathrm{i}}\right)+{\mathrm{d}}_3^{+}-{\mathrm{d}}_3^{-}=1$$

(33)

Variable bounds

11 ≤ X₁ ≤ 100 (K/min)

30 ≤ X₂ ≤ 100 (μm)

0.18 ≤ X₃ ≤ 0.3 (%)

0.7 ≤ X₄ ≤ 1.5 (%)

Bounds on deviation variables

$${\mathrm{d}}_{\mathrm{i}}^{-},{\mathrm{d}}_{\mathrm{i}}^{+}\ge 0{ \mathrm{and} \mathrm{d}}_{\mathrm{i}}^{-}{\ast \mathrm{d}}_{\mathrm{i}}^{+}=0 , \mathrm{i}=1,2,3$$

(34)

Minimize

Minimize the deviation function in the initial design

$$\mathrm{Z}={\sum }_{\mathrm{i}=1}^3{\mathrm{W}}_{\mathrm{i}}\left({\mathrm{d}}_{\mathrm{i}}^{-}+{\mathrm{d}}_{\mathrm{i}}^{+}\right); {\sum }_{\mathrm{i}=1}^3{\mathrm{W}}_{\mathrm{i}}=1$$

(35)

There are three goals in the problem—(i) minimize ferrite grain size (D_α), (ii) maximize ferrite Fraction (X_f), and (iii) minimize interlamellar spacing (S_o). The targets and the acceptable values of the three goals are given. All three goals are nonlinear. There are four design/system variables—cooling rate (CR), final austenite grain size after rolling (D), the carbon concentration ([C]), and the manganese concentration after rolling ([Mn]). We aim to obtain:

• The range of the system variables to reach the target of each goal of the best RMC for different design preferences and
• The weight set, ${\mathrm{W}}^{\mathrm{s}}={{\displaystyle \cap }}_{\mathrm{k}\in \mathrm{K}}{\mathrm{W}}_{\mathrm{k}}^{\mathrm{s}}$, that is a compromise of the three goals for different design preferences.

In [36], the authors formulate and execute the initial compromise decision support problem for the hot-rolling process chain problem and carry out a weight sensitivity analysis to identify variable values and the weight set. Ternary plots are generated to visualize and explore the weight set. In each ternary plot, the three axes represent the weights assigned to the three goals, respectively, and the contours indicate the fulfillment of each goal, $\frac{{\mathrm{G}}_{\mathrm{k}}}{{\mathrm{T}}_{\mathrm{k}}}$, $\mathrm{k}=1, 2, 3$. Since the goals conflict, compromise solutions are desired. Weight sensitivity analysis is a way to mediate compromise around the conflicts among the goals. ${\mathrm{W}}_1^{\mathrm{s}}$, ${\mathrm{W}}_2^{\mathrm{s}}{ \mathrm{W}}_3^{\mathrm{s}}$, and ${\mathrm{W}}^{\mathrm{s}}$ are the satisficing weight regions identified in the ternary plots, see areas marked using arrows in Figure 11a–d, respectively. To compare $\frac{{\mathrm{G}}_{\mathrm{k}}}{{\mathrm{T}}_{\mathrm{k}}}$, $\mathrm{k}=1, 2, 3$ at the same scale, we normalize the fulfillments of each goal under all weight scenarios in the range [0, 1]. An acceptable fulfillment $\frac{{\mathrm{G}}_{\mathrm{k}}}{{\mathrm{T}}_{\mathrm{k}}}$ of each goal is the dashed line in each ternary plot. The area between the corner (best $\frac{{\mathrm{G}}_{\mathrm{k}}}{{\mathrm{T}}_{\mathrm{k}}}$) and the dashed line (acceptable $\frac{{\mathrm{G}}_{\mathrm{k}}}{{\mathrm{T}}_{\mathrm{k}}}$) is the satisficing weight area of Goal $\mathrm{k}$, ${\mathrm{W}}_{\mathrm{k}}^{\mathrm{s}}$ and the superimposed area is the satisficing weight set of all three goals, W^s.

The steps in the RMC tuning algorithm as applied to the hot rod-rolling problem are given in Algorithm A1, Appendix A. In Figure 11d, Figure 12d and Figure 13d, the satisficing weight set W^s when the RMC is 0.1, 0.5, and 0.8 is shown, respectively. When RMC is 0.1, as Figure 11d shows, W^s is large, whereas when RMC is 0.8, as Figure 13d shows, W^s is small. However, using the ALP, there is no mechanism to evaluate which RMC value results in a relatively accurate approximation that gives us a robust W^s. So, we fill this gap using ALPPL.

4.2. Parameter Initialization

There are nineteen weight scenarios used in

Table 7. Weight vectors used in [36] as different design scenarios.

	W1	W2	W3		W1	W2	W3
1	1	0	0	11	0	0.75	0.25
2	0	1	0	12	0	0.25	0.75
3	0	0	1	13	0.33	0.33	0.33
4	0.5	0.5	0	14	0.2	0.2	0.6
5	0.5	0	0.5	15	0.4	0.2	0.4
6	0	0.5	0.5	16	0.2	0.4	0.4
7	0.25	0.75	0	17	0.6	0.2	0.2
8	0.25	0	0.75	18	0.4	0.4	0.2
9	0.75	0	0.25	19	0.2	0.6	0.2
10	0.75	0.25	0

, representing a variety of design preferences.

Running Processes B1 and B2 using the sample RMC (0.1, 0.5, and 0.8), we obtain the EIs in

Table 8. Results of EIs using sample RMC values with nineteen design scenarios.

RMC	Statistics	Z	Nit	Nacc	Nab	Naoc
0.1	μ	0.1480	46.58	18.74	1.79	0.84
	σ	0.0679	5.95	0.87	0.63	0.50
0.5	μ	0.1467	20.42	19.47	2.05	0.79
	σ	0.0675	9.47	0.84	0.62	0.42
0.8	μ	0.1480	8.32	14.16	2.21	0.79
	σ	0.0675	5.56	6.94	0.63	0.71

.

Running Process B3, we obtain the initial RMC (${\mathrm{RMC}}_0$) and the best RMC as 0.5 (its ${\mathsf{\mu }}_{\mathrm{Z}}$ and ${\mathsf{\sigma }}_{\mathrm{Z}}$ are bold italic), and we initialize the DEI as shown in

Table 9. The initial DEIs.

DEI of μz	DEI of σz	DEI of μNaoc	DEI of σNaoc	DEI of μNab	DEI of σNab
[0, 0.1477]	[0, 0.0677]	[0, 0.82]	[0, 0.55]	[0, 1.95]	[0, 0.63]

. The results of the parameter initialization—RMC₀ (0.5), EIs (Table 5), DEIs (Table 9), and “best” RMC value 0.5—are the input of RMC tuning (C).

4.3. RMC Tuning

We make rules for each procedure of RMC tuning based on heuristics. The heuristics are generalized from parameter learning and can be adjusted through the search process.

C3: Evaluate the result of current RMC based on EIs and DEI.

C3-1: Determine the next RMC value.

Rule 1: Compare the performance of multiple EIs and define the comparison rules. Lines 22–30 in Algorithm A1 in Appendix A are an expansion of this rule. RMC A is better than RMC B because no less than κ of the EI(A) are better than EI(B), whereas other EI(A) do not exceed γ of the upper and lower bound of DEI. In this problem, we set κ = 1/2 and γ = 3 0%.

Rule 2: Determine when and how the RMC should be updated. Lines 6–13 in Appendix A explain this rule. We use a hill-climbing approach to update the RMC. If the updating in the previous RMC-tuning cycle improves the performance, then the previous updating is in the “hill-climbing direction,” and we continue updating the RMC in this direction with a step size α; otherwise, we need the best RMC to “pull us back” to the right direction with a portion β; hence, we update the RMC as a linear combination of the best RMC (elite) and the RMC two cycles ago (parent). In this problem, we set α as a random value in [0, 1] and β as a random value in [0.5, 1]. In this way, we incorporate greediness, elitism, and randomness in evolution.

C3-2: Evaluate whether DEI needs to be updated.

Rule 3: Determine when and how the DEI should be updated. See Algorithm A1 in Appendix A, Lines 18–38. In an RMC-tuning cycle, if more than κ EIs are better than the previous cycle, and more than $\mathrm{\iota }$ EIs are in the DEI, whereas no more than γγ EIs have minor violations, then the DEI is updated. Here, we set $\mathsf{\kappa }=\mathrm{\iota }=2/3$. For a problem with more EIs, κ and $\mathrm{\iota }$ can be tuned using the performance improvement rate. This rule prevents an over restrictive DEI from blocking us from a better range while ensuring gradual and relatively conservative updating of the DEI.

C3-3: Evaluate whether “the best RMC” needs to be updated.

Rule 4: Determine when and how the best RMC is updated. See Algorithm A1 in Appendix A, Lines 34–36. We use a variable (“best”) to store the best RMC. If more than κ EIs of the current RMC are better than the EIs of the best, the current RMC becomes the new best.

C3-4: We aggregate the results of the RMC tuning RMC _t+1, DEI, and best as inputs for the next tuning iteration.

C4: Determine whether to stop iterating.

Rule 5: Make the stopping criteria. To stop RMC tuning at the appropriate time, we use “the maximum number of RMC-tuning iterations” and “the maximum number of RMC-tuning iterations without updating the best RMC” as the stopping criteria.

4.4. Results

Using our test problem, the RMC tuning stops after fourteen iterations. We identify 0.55 as the best RMC for the test problem. Compared with the initial RMC of 0.5, the final best RMC of 0.55 improves ${\mathsf{\sigma }}_{\mathrm{Z}}$, ${\mathsf{\sigma }}_{\mathrm{Naoc}}$, and ${\mathsf{\sigma }}_{\mathrm{Nab}}$. The RMCs and EIs in the fourteen iterations are in

Table 10. The EIs, DEI, RMC, best RMC of the fourteen iterations of RMC tuning.

Iteration	RMC	Weight-Sum Deviation		Number of Active Original Constraints		Number of Active Bounds		Better Than Cycle (t-1)	Better Than Best RMC	Update DEI
		μz	σz	μNaoc	σNaoc	μNab	σNab
1	0.5	0.147	0.068	0.79	0.42	2.05	0.62	-	-	[1, 1.95] -> [1, 2.05]
2	1.0	0.152	0.071	0.95	0.78	2.26	0.45	N	N	-
3	0.8	0.148	0.068	0.79	0.71	2.21	0.63	N	N	-
4	0.6	0.147	0.067	0.95	0.52	2.00	0.58	Y	Y	->
5	0.4	0.147	0.068	0.68	0.48	2.00	0.58	Y	Y	-
6	0.2	0.147	0.068	0.84	0.60	1.84	0.60	N	N	->
7	0.3	0.154	0.069	0.95	0.52	1.79	0.71	N	N	-
8	0.45	0.147	0.068	0.89	0.66	2.00	0.58	Y	N	-
9	0.55	0.147	0.067	0.84	0.37	2.05	0.52	Y	Y	-
10	0.65	0.147	0.067	1.00	0.58	2.05	0.62	N	N	->
11	0.48	0.147	0.068	0.89	0.66	2.05	0.62	N	N	-
12	0.53	0.147	0.067	0.84	0.69	2.00	0.58	Y	N	-
13	0.57	0.147	0.067	0.84	0.37	2.11	0.57	N	N	-
14	0.43	0.145	0.069	0.84	0.60	2.00	0.58	N	N	-

. The DEI is updated four times, and the best RMC is updated three times. The final best RMC is in the ninth iteration.

4.5. Verification and Validation

To verify the efficacy of ALPPL, we evaluate its adequacy and the necessity in returning robust solutions.

Adequacy—We tested 20 RMC values randomly spread in [0, 1] and identified the relatively insensitive range(s); see Figure 14. If the best RMC returned by the ALPPL falls into the relatively insensitive range(s), then we show that the ALPPL can identify the best RMC. It turns out that the best RMC (0.55) ensures the solutions to fall in a relatively insensitive range. For each pair of EIs (the mean and standard deviation), we identify the range(s) of RMC values, within which, both the mean and the standard deviation have acceptable values and are flat (stable). For example, for the deviation Z, we desire both its mean μ_z and standard deviation σ_z to be low values and flat. The two ranges bounded by the dotted rectangles are such ranges. For all EIs, the overlapped desired range is [0.5, 0.55], so this is the relatively insensitive range. The value of 0.55 is in the insensitive RMC range for all EIs, so it is verified that when RMC is 0.55, it gives a relatively robust performance.

Necessity—The insensitive range is sufficiently explored during the RMC tuning. First, we identify the insensitive range of RMC values—testing 20 RMC values uniformly distributed in [0, 1] with their EIs, we identify two ranges where the solutions are relatively insensitive to scenario changing, [0.35, 0.4] and [0.5, 0.55]. In Figure 15a, we illustrate all fourteen RMC values in RMC tuning. The horizontal axis represents the iteration number and the vertical axis represents the RMC value. Four of the fourteen RMC values fall in the two insensitive ranges, so 28.5% of the tested RMC values fall in the insensitive ranges, whereas the insensitive ranges only occupy 10% of the whole RMC range. Hence, we conclude that our RMC tuning enables a relatively sufficient exploration of the insensitive ranges.

Validation of the improvement of ALPPL versus the ALP. We compare the tested RMC values using parameter learning (ALPPL) in Figure 15a with the tested RMC values using the golden section search (ALP) in Figure 15b. The best RMC identified using the golden section search is 0.65, which is not in the insensitive range; in addition, the RMC values tested in the golden section search are concentrated in [0.57, 0.77], which misses the insensitive ranges [0.35, 0.4] and [0.5, 0.55]. For a solution algorithm that sequentially linearizes the problems in multiple synthesis cycles, the computational complexity is O(n²) [37]—this applies to both the ALP and ALPPL, so the parameter-learning method in this paper does not require much computational power. Other improvements of ALPPL over ALP are summarized in

Table 11. ALPPL with RMS tuning versus ALP with golden section search.

		ALPPL	ALP
General comparison	Search method	Rule-based parameter learning	Golden section search
	Criteria used for evaluation of the RMC	Deviation (fulfillment of the goals), the robustness of the solution	Fulfillment of the goals
	If the approximation is sensitive to the scenario changing	Considering different scenarios, the most appropriate RMC is identified. The approximation is relatively insensitive to scenario changes	In each scenario, the best RMC is identified; it may vary as the scenario changes. The approximation is relatively sensitive to scenario changes
	Stopping criteria	The best RMC has not been updated for n iterations, or the total iteration number reaches a threshold	The distance between two golden section points is less than a threshold $\epsilon$
	Complexity	O(n2)	O(n2)
Comparison of the cooling problem results	Number of search iterations	14	12
	If the identified best RMC is in the insensitive range	Yes	No
	Number of tested RMC values falling into insensitive range	4	2
	Is the insensitive range explored sufficiently	Relatively sufficiently	Insufficiently

.

5. Closing Remarks

Our aim in this paper is to improve the determination of one critical parameter in a satisficing method regarding the robustness of the solution and the accuracy of the linearization. We show the benefits of using a satisficing strategy, the cDSP, and ALP to manage engineering design problems, especially when there are multiple goals, nonlinear and nonconvex equations, and the goals have different levels of achievability. However, there is a drawback of the current ALP—one of its parameters, the RMC, has an impact on the linearization and the solution space. For years, the RMC was either user-defined or determined using golden section search. However, there was no mechanism for obtaining insight to improve the approximation by controlling the RMC. The golden section approach may result in missing the sub-range of the RMC with good approximation performance. The best RMC value is sensitive to design scenario changes; only the weighted-sum deviations and feasibility of the solutions are considered when evaluating approximation performance. Hence, to improve the ALP, we incorporate parameter learning into the algorithm and upgrade it to the adaptive linear programming algorithm with parameter learning algorithm (ALPPL). In the ALPPL, we improve the approximation performance in three steps—identifying the criteria for approximation performance, developing evaluation indices (EIs), and tuning the RMC. We use an industry-inspired problem, namely, the hot-rolling process for a steel rod, to demonstrate the improvements of ALPPL (to access DSIDES and ALPPL, please contact the corresponding author. The tutorial for using DSIDES to formulate a cDSP is in this video: https://www.youtube.com/watch?v=tUpVC97Y1L8, accessed on 25 November 2020) over the ALP. The computational complexity of ALPPL is the same as the ALP, which is O(n²).

The parameter-learning method used in this paper can be expanded to other algorithms that apply heuristics to determine the value of parameters. Domain expertise can be helpful in identifying the criteria for performance evaluation and developing evaluation indices. With domain knowledge, the computational complexity of the parameter-learning algorithms can be well controlled within an inexpensive range. This can customize the parameter learning for each algorithm and each problem.

One limitation of this paper is that we did not discuss in great detail the computational complexity of the ALPPL or the parameter learning for algorithm improvement in general, because it can vary for different algorithms or problems. One thing that we can ensure is that, with designers’ domain expertise, by identifying the most critical criteria and then quantifying them as the most representative evaluation indices, the computational power required for the proposed parameter-learning algorithm can be less than that for parameter tuning without domain expertise, since designers can remove the less relevant criteria and insensitive evaluation indices.

In future research, there are several promising directions to explore in the realm of engineering design using the satisficing strategy. Firstly, investigating the applicability and effectiveness of the proposed rule-based parameter-learning procedure across a broader range of engineering problems could provide valuable insights into its generalizability and robustness. Additionally, exploring alternative satisficing algorithms or enhancements to existing algorithms could further improve the efficiency and robustness of design processes. Furthermore, incorporating advanced machine learning techniques, such as reinforcement learning or neural networks, into the satisficing framework may offer new opportunities for adaptive and intelligent decision making in engineering design. Finally, conducting empirical studies and applications in diverse industries and contexts could validate the practical value and benefits of the proposed approach, thereby facilitating its adoption and integration into engineering practice. Overall, these future research directions hold the potential to advance the state of the art in satisficing engineering design and contribute to the development of more automated and robust design methods.

6. Glossary

Accumulated constraint(s): When a nonlinear constraint is concave (the degree of the convexity is lower than −0.015), the approximated linear constraint of the previous iteration is carried forward and combined with its approximated linear constraint in the current iteration. Thus, a concave nonlinear constraint is represented by multiple linear constraints, and those carried forward are called accumulated constraints.

Active constraint: If the slack or surplus of a constraint at a solution point is zero, then it is an active constraint.

Convexity degree: The average value of the diagonal terms of the Hessian matrix of a function. This definition is applied to any paper on the cDSP and ALP.

Deviation: The percentage difference between the value of a goal at a solution and its target.

DSIDES: DSIDES is short for decision support in the design of engineering systems, which is a computational platform for designers to formulate engineering design problems using the compromise decision support problem (cDSP) and linearize and solve the problems using the adaptive linear programming (ALP) algorithm.

Parameter learning: This includes a set of activities to maximize the approximation performance—identifying the evaluation indices (EIs), the desired range of each evaluation index (DEI), and the value of a parameter, namely, the reduced move coefficient (RMC), that makes each EI fall into its desired range, DEI.

Parameter tuning: This means finding the value of a parameter, namely, the reduced move coefficient (RMC), that makes each EI fall into its desired range, DEI. It is a part of parameter learning.