Generalized Benders Decomposition Method to Solve Big Mixed-Integer Nonlinear Optimization Problems with Convex Objective and Constraints Functions

Abstract

The paper presents the Generalized Benders Decomposition (GBD) method, which is now one of the basic approaches to solve big mixed-integer nonlinear optimization problems. It concentrates on the basic formulation with convex objectives and constraints functions. Apart from the classical projection and representation theorems, a unified formulation of the master problem with nonlinear and linear cuts will be given. For the latter case the most effective and, at the same time, easy to implement computational algorithms will be pointed out.

1. Introduction

In the early 1960s, a Dutch mathematician Jacques F. Benders was considering optimization problems of the form [1]:

$$\underset{x,v}{max}{c}^{T}x+f\left(v\right)$$

(1)

subject to the following constraints (hereinafter, the abbreviation “s.t.” will be used):

$$Ax+F\left(v\right)\le b,b\in {\mathbb{R}}^m$$

(2)

$$x\in X\subseteq {\mathbb{R}}^n,v\in V\subseteq {\mathbb{R}}^q$$

(3)

He called them mixed-variables programming problems. In these problems, both the objective functions and the functions defining constraints were sums of two components: linear, dependent on the vector variable $x\in X$, and nonlinear, dependent on the vector variable $v\in V$. Benders called v a vector of complicating variables, because when they were fixed, the problem simplified—it became linear. In addition, he suggested that these variables can be discrete, i.e., the problem with respect to the x vector itself became continuous and was already very easy to solve with any linear programming solver (e.g., based on the simplex algorithm).

In his work [1] Benders proposed an iterative solution procedure for this optimization problem, consisting in solving alternately auxiliary problems with respect to one or the other vector of variables. These problems were related to the dual representation of the initial problem and optimality conditions. We will call the problem with respect to complicating variables v the master problem, and the problem with respect to x variables the primal problem. In its subsequent launches the primal problem provided the master problem with constraints related to the approximation of the objective function or the feasible set, computed at different points of the X set. The former were added in case, when for a given trial vector v, there existed a solution admissible in the initial problem, the latter—otherwise.

The generalization of this approach to nonlinear problems was presented by Arthur Geoffrion [2]. He called it the Generalized Benders Decomposition (GBD). As in the original work by Benders, constraints in the master problem were defined with the help of Lagrangians of the respective primal problems. The objective values in optimal points, obtained after each iteration of the master problem and of the primal problem—when it was feasible, were used to estimate the optimal objective of the initial problem. In the case of a minimization problem, the master problem provided its estimate from the bottom $LBD$ (from the Lower BounD), the primal problem—from the top—$UBD$ (from the Upper BounD). When the primal problem was infeasible, the so-called feasibility problem, consisting in minimization of the maximum exceedance of constraints, was solved. The stop test consisted in checking whether the upper and the lower estimates of the objective function were equal with a given accuracy.

Geoffrion’s approach in a slightly modified version was presented in the works of Floudas [3,4], which assumed that v variables are binary and supplemented the constraints of the problem with the equality ones. Unfortunately, they are not relaxed in the feasibility problem (like inequality constraints), which can cause this approach to lead to a deadlock, when for a trial point selected from the set V, any of these constraints is not satisfied for all $x\in X$. Floudas provided several versions of the computing algorithm, but they concerned the same specific problems, actually being one algorithm written in different ways.

Unfortunately, neither the works of Floudas [3,4], nor the Geoffrion’s source work [2], presented a version of fundamental importance in convex problems with respect to the entire vector $(x,v)$—with linear cuts in a master problem. Despite it being the easiest version to implement; moreover, it was mentioned by Benders [1] and in more recent works on mixed integer nonlinear programming, [5,6] it is most often presented as the only one.

Deficiencies of the above-mentioned presentations of the Benders method and the lack of a coherent and uniform description of this approach convinced the author of the need to put this knowledge in order, which resulted in this work. The importance of this approach is growing as more and more practical problems of mixed-integer nonlinear programming are solved, related to, e.g., energy [7], telecommunications [8], transport [9,10], gas networks [11], production processes [12], water systems [13], two-stage nonlinear stochastic control [14,15]. When these problems exceed a certain size, even the best commercial solvers fail [14]. It is necessary to write your own specialized code, using structural properties of the problem. That is what the Benders method is great for. It is also a kernel of many complicated algorithms solving optimization problems with nonconvex functions, especially in the field of chemical engineering [16,17,18,19,20].

Generalized Benders Decomposition is not the only decomposition method to solve MINLP problems. The most popular other classical approaches are: Lagrangian relaxation, branch and bound, column generation, outer approximation [6,21]. Recently, the Alternating Directions Multiplier Method has been gaining a lot of popularity [22,23].

2. Problem Formulation

Let us consider an optimization problem of the following form:

$$\underset{x,v}{min}f(x,v)$$

(4)

s.t.

$$g(x,v)\le 0$$

(5)

$$x\in X\subseteq {\mathbb{R}}^n$$

(6)

$$v\in V\subseteq {\mathbb{R}}^q$$

(7)

where

$$\begin{array}{cc}\hfill f:& {\mathbb{R}}^n\times {\mathbb{R}}^q\to \mathbb{R},\hfill \\ \hfill g:& {\mathbb{R}}^n\times {\mathbb{R}}^q\to {\mathbb{R}}^m\hfill \end{array}$$

The possible equality constraints

$$h_j(x,v)=0,j=1,\dots ,s$$

(8)

can be taken into account using standard transformations, e.g.,

$$\left\{\begin{array}{c}{h}_1(x,v)\le 0\\ ⋮\\ h_s(x,v)\le 0\\ \\ \sum _{j=1}^s{h}_j(x,v)\ge 0\end{array}\right.$$

(9)

or

$$h_j^2(x,v)\le 0,j=1,\dots ,s$$

(10)

For the sake of simplicity, the equality constraints will be further omitted. We will mention them in Section 8, suggesting a specific way to treat them in computational algorithms.

Assumptions about the functions and sets appearing in the formulation will be given in the following statements. For now, we will only assume, that they are such that the solution of the problem (4)–(7) exists. It is only worth saying something about the V set. It is supposed to be contained in ${\mathbb{R}}^q$. Therefore, this formulation covers also MINLP problems (from Mixed Integer NonLinear Programming), that is mixed continuous–discrete, nonlinear problems, where $V\subseteq {\mathbb{Z}}^q$; $\mathbb{Z}$—the space of all integers.

It is suggested to define the vector v—complicating variables in such a way that, when we establish it, one of the following occurs:

• A problem can be decomposed into a number of independent subproblems, each using a different subproblem vector $x_i,i=1,\dots ,p$

  $$x=\left[\begin{array}{c}{x}_1\hfill \\ x_2\hfill \\ ⋮\hfill \\ x_p\hfill \end{array}\right]$$

  (11)
  Most often, the total objective function is additive:

  $$\underset{x,v}{min}f(x,v)=\underset{x_1,x_2,\dots ,x_p,v}{min}\sum _{i=1}^p{f}_i(x_i,v)+f_0\left(v\right)$$

  (12)

  s.t.

  $$g_{ij}(x_i,v)\le 0,j=1,\dots ,m_i;i=1,\dots ,p$$

  (13)

  $$x_i\in X_i\subseteq {\mathbb{R}}^{n_i},i=1,\dots ,p,v\in V\subseteq {\mathbb{R}}^q,$$

  (14)

  $$\sum _{i=1}^p{n}_i=n$$
  In this case, it is worthwhile to use not only decomposition, but also a parallel solution to the resulting primal subproblems.
• The problem with respect to x variables takes a specific structure for which efficient algorithms exist—e.g., it is linear or quadratic.
• The problem is convex with respect to x with v fixed and vice-versa, although it is non-convex, if we consider it for the concatenated vector variable $(x,v)$.
  Convex problems with respect to one subvector when the latter is fixed, for example, bilinear problems, are very often formulated in engineering [12,24].

3. Examples of Optimization Problems That Can Be Solved with the Benders Method

• Problem with a separable objective function and separable function constraints

  $$\underset{y_1,\dots ,y_6}{min}\left[2y_1^2-y_1·y_2+4y_2^2+{(y_3-4)}^2+{(y_4-3)}^2+8y_5^2+y_6^2-3y_5\right]$$

  (15)

  s.t.

  $$y_1\ge 1,y_2\ge 0,y_3\ge 3,y_4\ge 2,y_5\ge 3,y_6\ge 0$$

  (16)

  $$y_1+y_2+y_3^2+y_4^2+y_5+y_6^2\le 50$$

  (17)
  In problems, where both the objective function and the constraint functions are sums of functions dependent on subvectors of decision variables, to obtain the standard form (12)–(14) you need to accordingly group the variables into subvectors and to introduce additional v variables, separating constraints.
  Let us assume $p=$ 3 and denote:

  $$x_1=\left[\begin{array}{c}{x}_{1,1}\\ x_{1,2}\end{array}\right]\triangleq \left[\begin{array}{c}{y}_1\\ y_2\end{array}\right],x_2=\left[\begin{array}{c}{x}_{2,1}\\ x_{2,2}\end{array}\right]\triangleq \left[\begin{array}{c}{y}_3\\ y_4\end{array}\right],$$

  (18)

  $$x_3=\left[\begin{array}{c}{x}_{3,1}\\ x_{3,2}\end{array}\right]\triangleq \left[\begin{array}{c}{y}_5\\ y_6\end{array}\right]$$
  Now:

  $$f_1\left(x_1\right)=2x_{1,1}^2-x_{1,1}{x}_{1,2}+4x_{1,2}^2$$

  (19)

  $$X_1=\left\{[x_{1,1},x_{1,2}]\in {\mathbb{R}}^2:x_{1,1}\ge 1,x_{1,2}\ge 0\right\}$$

  (20)

  $$f_2\left(x_2\right)={(x_{2,1}-4)}^2+{(x_{2,2}-3)}^2$$

  (21)

  $$X_2=\left\{[(x_{2,1},x_{2,2}]\in {\mathbb{R}}^2:x_{2,1}\ge 3,x_{2,2}\ge 2\right\}$$

  (22)

  $$f_3\left(x_3\right)=8x_{3,1}^2+x_{3,2}^2-3x_{3,1}$$

  (23)

  $$X_3=\left\{[x_{3,1},x_{3,2}]\in {\mathbb{R}}^2:x_{3,1}\ge 3,x_{3,2}\ge 0\right\}$$

  (24)

  $$f_0\left(v\right)\equiv 0$$

  (25)
  Let us also introduce artificial variables $v_1,v_2,v_3$ for constraining components binding primal subvector variables, respectively: $x_1,x_2$ and $x_3$, in the cumulative constraint (17).

  $$x_{1,1}+x_{1,2}\le v_1$$

  (26)

  $$x_{2,1}^2+x_{2,2}^2\le v_2$$

  (27)

  $$x_{3,1}+x_{3,2}^2\le v_3$$

  (28)
  We obtain the following constraint functions in the problem (12)–(14) format:

  $$g_{11}(x_1,v)=x_{1,1}+x_{1,2}-v_1$$

  (29)

  $$g_{21}(x_2,v)=x_{2,1}^2+x_{2,2}^2-v_2$$

  (30)

  $$g_{31}(x_3,v)=x_{3,1}+x_{3,2}^2-v_3$$

  (31)

  as well as the V set:

  $$V=\left\{[v_1,v_2,v_3]\in {\mathbb{R}}^3:v_1+v_2+v_3\le 50\right\}$$

  (32)
• Problem with a chain (ring) of constraints.
  In problems of this type, successive constraints bind several subsequent decision variables, i.e., every variable appears in several constraints (at least in two) together with neighboring variables with a lower and a higher index. This structure is quite typical for optimal control problems after discretization of differential equations over time and replacing a resulting system of s nonlinear equations with $s+1$ inequality constraints using the transformation (9). The idea for splitting a problem and obtaining the standard form (12)–(14) is to treat some variables in a vector (e.g., those which divide it into several equal parts) as complicating variables $v_i$, and then the constraints in which these variables appear as mixed constraints of the type (13).
  Consider the following problem:

  $$\underset{y\in {\mathbb{R}}^9}{min}{y}_1^2+y_2^2+\dots +y_9^2$$

  s.t.

  $$y_{k+1}-y_k\le sink,k=1,...,8$$

  $$y_1-y_9\le 0.5$$
  Let us denote:

  $$v_1\triangleq y_3,v_2\triangleq y_6,v_3\triangleq y_9$$
  The remaining elements will form subvectors of the dimension 2. We will denote them as $x_1,x_2,x_3$, that is:

  $$x_1=\left[\begin{array}{c}{x}_{1,1}\\ x_{1,2}\end{array}\right]\triangleq \left[\begin{array}{c}{y}_1\\ y_2\end{array}\right],x_2=\left[\begin{array}{c}{x}_{2,1}\\ x_{2,2}\end{array}\right]\triangleq \left[\begin{array}{c}{y}_4\\ y_5\end{array}\right],$$

  $$x_3=\left[\begin{array}{c}{x}_{3,1}\\ x_{3,2}\end{array}\right]\triangleq \left[\begin{array}{c}{y}_7\\ y_8\end{array}\right]$$
  These problems can be presented in the format (12)–(14) assuming:

  $$f_1\left(x_1\right)=x_{1,1}^2+x_{1,2}^2$$

  (33)

  $$X_1=\left\{[x_{1,1},x_{1,2}]\in {\mathbb{R}}^2:x_{1,2}-x_{1,1}\le sin1\right\}$$

  (34)

  $$g_{11}(x_1,v)=x_{1,1}-v_3-0.5$$

  (35)

  $$g_{12}(x_1,v)=v_1-x_{1,2}-sin2$$

  (36)

  $$f_2\left(x_2\right)=x_{2,1}^2+x_{2,2}^2$$

  (37)

  $$X_2=\left\{[x_{2,1},x_{2,2}]\in {\mathbb{R}}^2:x_{2,2}-x_{2,1}\le sin4\right\}$$

  (38)

  $$g_{21}(x_2,v)=x_{2,1}-v_1-sin3$$

  (39)

  $$g_{22}(x_2,v)=v_2-x_{2,2}-sin5$$

  (40)

  $$f_3\left(x_3\right)=x_{3,1}^2+x_{3,2}^2$$

  (41)

  $$X_3=\left\{[x_{3,1},x_{3,2}]\in {\mathbb{R}}^2:x_{3,2}-x_{3,1}\le sin7\right\}$$

  (42)

  $$g_{31}(x_3,v)=x_{3,1}-v_2-sin6$$

  (43)

  $$g_{32}(x_3,v)=v_3-x_{3,2}-sin8$$

  (44)

  $$f_0\left(v\right)=\sum _{i=1}^3{v}_i^2$$

  (45)

4. Decomposition

The problem (4)–(7) can be represented as follows [2]:

$$\underset{v\in V}{min}\underset{x\in X}{inf}f(x,v)$$

(46)

s.t.

$$g(x,v)\le 0$$

(47)

The infimum in the internal problem appeared due to the fact that for some v it can be unbounded.

Let us denote

$$z\left(v\right)\triangleq \underset{x\in X}{inf}f(x,v)$$

(48)

s.t.

$$g(x,v)\le 0$$

(49)

We will call the problem (48)–(49), solved for a fixed v, the primal problem.

The problem (46)–(47) can now be written as

$$\underset{v\in V\cap V_0}{min}z\left(v\right)$$

(50)

where

$$V_0=\left\{v:\exists x\in Xg(x,v)\le 0\right\}$$

(51)

and interpreted as a projection of the problem (46)–(47) onto the space of the variable v [2]. The formulation (50) is called the master problem.

The requirement for v defined in (50) as $v\in V\cap V_0$ results from the necessity to guarantee the existence of a solution—the value of $z\left(v\right)$. The set $V_0$ is called the solvability set.

The problem is that we only know $z\left(v\right)$ and $V_0$ indirectly, through their generic definitions.

The following theorems are valid [2]:

Theorem 1

(Projection).

• The problem (4)–(7) has no solution or is unbounded if and only if the same is true for the problem (50).
• If $(\widehat{x},\widehat{v})$ is the optimal solution of the problem (4)–(7), then $\widehat{v}$ is the optimal solution of the problem (50),
• If $\widehat{v}$ is the optimal solution of the problem (50) and $\widehat{x}$ reaches the infimum in the problem (48)–(49) at $v=\widehat{v}$, then $(\widehat{x},\widehat{v})$ is the optimal solution of the problem (4)–(7).

Theorem 2

(Representation of $V_0$).

Assume that X is a nonempty convex set and that the function g is convex on X for each fixed $v\in V$.

Suppose further that the set

$$Z_v=\left\{z\in {\mathbb{R}}^m:\exists x\in Xg(x,v)\le z\right\}$$

is closed for each fixed $v\in V$. Then a point $v^{*}\in V$ also belongs to the set $V_0$ if and only if:

$$\underset{x\in X}{inf}{L}_f(x,v^{*},\lambda )\le 0,\forall \lambda \in \Lambda$$

(52)

where

$$\Lambda =\left\{\lambda \in {\mathbb{R}}^m:\lambda \ge 0,\sum _{j=1}^m{\lambda }_j=1\right\}$$

(53)

and

$$L_f(x,v,\lambda )={\lambda }^{T}g(x,v)$$

(54)

Theorem 3

(Representation of $z\left(v\right)$).

Assume that X is a nonempty convex set and that the functions f and g are convex on X for each fixed $v=v^{*}\in V$. Assume further that for $v^{*}$, at least one of the following conditions is met:

• $z\left(v^{*}\right)$ is finite and in the problem (48)–(49) there exists an optimal vector of Lagrange multipliers;
• $z\left(v^{*}\right)$ is finite, $g(x,v^{*})$ and $f(x,v^{*})$ are continuous on X, the set X is closed and the set of optimal solutions of the problem (48)–(49) with an accuracy ε is nonempty and bounded for some $\epsilon \ge 0$;

Then

$$z\left(v\right)=\underset{\lambda \ge 0}{sup}\underset{x\in X}{inf}{L}_o(x,v,\lambda ),\forall v\in V\cap V_0$$

(55)

where

$$L_o(x,v,\lambda )=f(x,v)+{\lambda }^{T}g(x,v)$$

(56)

The last theorem results directly from the strong duality theorem [25].

Substituting in the problem (50) the expression (55) for $z\left(v\right)$ and (52) for the $v\in V_0$ constraint, we obtain an equivalent problem:

$$\underset{v\in V}{min}\underset{\lambda \ge 0}{sup}\underset{x\in X}{inf}{L}_o(x,v,\lambda )$$

(57)

s.t.

$$\underset{x\in X}{inf}{L}_f(x,v,\lambda )\le 0,\lambda \in \Lambda$$

(58)

Using the definition of supremum as the smallest upper constraint and introducing an additional, scalar variable $\mu$, we obtain the following form of the master problem, equivalent to (57)–(58):

$$\underset{v\in V,\mu }{min}\mu$$

(59)

s.t.

$$\underset{x\in X}{inf}{L}_o(x,v,\lambda )\le \mu ,\forall \lambda \ge 0$$

(60)

$$\underset{x\in X}{inf}{L}_f(x,v,\lambda )\le 0,\forall \lambda \in \Lambda$$

(61)

In practice, it can be assumed that the function $z\left(v\right)$ (see formula (48)) is constrained for all $v\in V$, set X compact and functions $f(x,v)$ and $g(x,v)$ are continuous throughout the domain. Therefore, we can replace the infimum with minimum. Then the master problem will take the following form:

$$\underset{v\in V,\mu }{min}\mu$$

(62)

s.t.

$$\underset{x\in X}{min}{L}_o(x,v,\lambda )\le \mu ,\forall \lambda \ge 0$$

(63)

$$\underset{x\in X}{min}{L}_f(x,v,\lambda )\le 0,\forall \lambda \in \Lambda$$

(64)

where the $L_o$ and $L_f$ functions are given by formulas (54) and (56).

The problem (62)–(64) is very difficult to solve due to constraints, which have to be satisfied in an infinite and even uncountable number of points (for all $\lambda$ with nonnegative coordinates or nonnegative summing up to unity) and the existence of internal optimization subproblems (minimization with respect to x).

These difficulties are overcome by relaxing this problem, more precisely by solving it with the use of successive, more and more precise, approximations of functions on the left hand side of the (63) and (64) constraints. They will be made up of pieces of the $L_o$ or $L_f$ functions related to the optimal solutions of the primal problem, in a situation, respectively, when the next trial point from the master problem $v^{*}$ belongs to $V_0$ or not.

5. Basic Properties of the Primal Problem

The primal problem is the initial problem (4)–(7) solved for a fixed $v=v^{*}\in V$

$$\underset{x\in X}{min}f(x,v^{*})$$

(65)

s.t.

$$g(x,v^{*})\le 0$$

(66)

While solving this problem, there may be two cases: when the primal problem is feasible, i.e., it has a solution, and when it has no solution.

5.1. The Case When the Primal Problem Has a Solution

Suppose that for $v^{*}$ obtained from the relaxed master problem, the primal problem (65)–(66) has a solution, to which corresponds the optimal vector of Lagrange multipliers ${\lambda }^{*}$. These multipliers will be the multipliers of the new constraint of type (63).

So a constraint of type

$$\underset{x\in X}{min}{L}_o(x,v,{\lambda }^{*})\le \mu$$

(67)

should be added to the relaxed master problem.

5.2. The Case When the Primal Problem Has No Solution

A more complicated situation occurs when the primal problem has no solution. According to Theorem 2, for a fixed $v=v^{*}$ the primal problem has a solution if the following condition is met:

$$\underset{x\in X}{min}\left[L_f(x,v^{*},\lambda )={\lambda }^{T}g(x,v^{*})\right]\le 0,\forall \lambda \in \Lambda$$

(68)

where

$$\Lambda =\left\{\lambda \in {\mathbb{R}}^m:\lambda \ge 0,\sum _{j=1}^m{\lambda }_j=1\right\}$$

(69)

If a solver detects that there is no solution of the primal problem, one should find a vector $\widehat{\lambda }\in \Lambda$ for which

$$\underset{x\in X}{min}{\widehat{\lambda }}^{T}g(x,v^{*})>0$$

(70)

Then it is taken that ${\lambda }^{*}=\widehat{\lambda }$ and the following inequality is added to the relaxed master problem:

$$\underset{x\in X}{min}{L}_f(x,v,{\lambda }^{*})\le 0$$

(71)

Theorem 4.

If the primal problem (65)–(66) is infeasible, the vector of Lagrange multipliers $\widehat{\lambda }\in \Lambda$ satisfying the condition (70) can be determined by solving the following auxiliary problem (the so-called feasibility problem):

$$\underset{x\in X}{min}\underset{j=1,\dots ,m}{max}{g}_j(x,v^{*})$$

(72)

Proof. 

The problem (72) is equivalent to the problem:

$$\underset{x\in X,\alpha }{min}\alpha$$

(73)

s.t.

$$g_j(x,v^{*})\le \alpha ,j=1,\cdots ,m$$

(74)

It certainly has a solution due to the previously taken assumptions concerning the $g(x,v)$ function.

Without loss of generality, let us assume that the set X is defined by the inequality constraints $r\left(x\right)$; that is, we have a problem:

$$\underset{x,\alpha }{min}\alpha$$

(75)

s.t.

$$g_j(x,v^{*})\le \alpha ,j=1,\cdots ,m$$

(76)

$$r_j\left(x\right)\le 0,j=1,\cdots ,t$$

(77)

We assume that all constraints functions g and r are continuously differentiable and convex, and the points of their activities are regular. The Lagrangian for this problem is as follows:

$$L_{\alpha f}(x,\alpha ,v^{*},\lambda )=\alpha +\sum _{j=1}^m{\lambda }_j^g·(g_j(x,v^{*})-\alpha )+\sum _{j=1}^t{\lambda }_j^r·r_j\left(x\right)$$

(78)

where ${\lambda }^g\ge 0,{\lambda }^r\ge 0$ are vectors of Lagrange multipliers for constraints (76) and (77), respectively, and $\lambda =\left[\begin{array}{c}{\lambda }^g\\ {\lambda }^r\end{array}\right]$. From the Karush–Kuhn–Tucker optimality conditions for the $\alpha$ variable we obtain the equation:

$$1-\sum _{j=1}^m{\widehat{\lambda }}_j^g=0$$

(79)

that is, comparing with the definition of $\Lambda$ (69), ${\lambda }^g\in \Lambda$. Taking into account the equality (79) in the Lagrangian formula (78) at the optimal point, as well as the conditions of complementarity in relation to the constraints (77), we will obtain:

$$L_{\alpha f}(x^{*},{\alpha }^{*},v^{*},\widehat{\lambda })=\sum _{j=1}^m{\widehat{\lambda }}_j^g·g_j(x^{*},v^{*})=L_f(x^{*},v^{*},{\widehat{\lambda }}^g)$$

(80)

So, the vector of Lagrange multipliers for the constraints (76) calculated at the optimal point in the problem (75)–(77) is the wanted $\widehat{\lambda }$ vector. □

Modern optimization solvers usually return such a vector of Lagrange multipliers when they state that a feasible solution does not exist.

6. Basic Properties of the Master Problem: Cuts

It would be best if, from the solutions of the primal problem for different $v=v^{*}$, it was possible to use not only Lagrange multipliers at the optimal point ${\lambda }^{*}$, but also optimal vectors of the primal variables $x^{*}$. To obtain a solution of the basic problem (4)–(7) using this approach, certain conditions must be met.

The most important is as follows:

Assumption 1.

Both Lagrangians $L_o(x,v,\lambda )$ and $L_f(x,v,\lambda )$ for any $x\in X$, $v\in V$ and $\lambda \ge 0$ can be written as composite functions:

$$L_o(x,v,\lambda )=Q_o(w_o(x,\lambda ),v,\lambda )$$

(81)

$$L_f(x,v,\lambda )=Q_f(w_f(x,\lambda ),v,\lambda )$$

(82)

where $w_o$, $w_f$ are scalar functions of x and λ, and $Q_o$, $Q_f$ are increasing functions of the first argument, convex of the second.

Theorem 5.

Let us suppose that for the problem (4)–(7) Assumption 1 is satisfied. Let $x^{*}$ be the optimal solution of the problem

$$\underset{x\in X}{min}{L}_o(x,v^{*},{\lambda }^{*})$$

(83)

Then also for $v\ne v^{*}$

$$\underset{x\in X}{min}{L}_o(x,v,{\lambda }^{*})=L_o(x^{*},v,{\lambda }^{*})$$

(84)

Proof. 

According to Assumption 1, due to the increasing dependency of the Lagrange function on the first argument, from the formula (81) we have:

$$Q_o(w_o(x^{*},{\lambda }^{*}),v^{*},{\lambda }^{*})=L_o(x^{*},v^{*},{\lambda }^{*})=\underset{x\in X}{min}{L}_o(x,v^{*},{\lambda }^{*})$$

$$=\underset{x\in X}{min}{Q}_o(w_o(x,{\lambda }^{*}),v^{*},{\lambda }^{*})=Q_o(\underset{x\in X}{min}{w}_o(x,{\lambda }^{*}),v^{*},{\lambda }^{*})$$

(85)

Hence, taking into account the injectivity of strictly monotone functions:

$$w_o(x^{*},{\lambda }^{*})=\underset{x\in X}{min}{w}_o(x,{\lambda }^{*})$$

(86)

So, for $v\ne v^{*}$

$$\underset{x\in X}{min}{L}_o(x,v,{\lambda }^{*})=\underset{x\in X}{min}{Q}_o(w_o(x,{\lambda }^{*}),v,{\lambda }^{*})=Q_o(\underset{x\in X}{min}{w}_o(x,{\lambda }^{*}),v,{\lambda }^{*})$$

$$=Q_o(w_o(x^{*},{\lambda }^{*}),v,{\lambda }^{*})=L_o(x^{*},v,{\lambda }^{*})$$

(87)

□

Consequently, if Assumption 1 is in effect, the constraint (63) in the master problem can be approximated around $v=v^{*}$ by the constraint:

$$\mu \ge L_o(x^{*},v,{\lambda }^{*})$$

(88)

where the vectors $x^{*}$, ${\lambda }^{*}$ come from the solution of the primal problem (65)–(66).

From now on this constraint will be called a cut.

From Assumption 1, yet another convenience follows. Given the convexity of the Lagrange function $L_o$ with respect to v, in the case when it is differentiable with respect to this vector, we obtain:

$$L_o(x^{*},v,{\lambda }^{*})\ge L_o(x^{*},v^{*},{\lambda }^{*})+\frac{\partial L_o^T}{\partial v}(x^{*},v^{*},{\lambda }^{*})(v-v^{*})$$

(89)

In this way, the cut (88) can be relaxed and replaced in the master problem by a linear cut

$$\mu \ge L_o(x^{*},v^{*},{\lambda }^{*})+\frac{\partial L_o^T}{\partial v}(x^{*},v^{*},{\lambda }^{*})(v-v^{*})$$

(90)

If the Lagrange function is not differentiable, the gradient can be replaced with a subgradient.

Analogical reasoning can be performed to the feasibility constraint (64).

It is easy to check that Assumption 1 is satisfied for two classes of problems:

• Separable—when the f, g functions are sums of components dependent on x and v, that is

  $$f(x,v)=f_1\left(x\right)+f_2\left(v\right)$$

  (91)

  $$g(x,v)=g_1\left(x\right)+g_2\left(v\right)$$

  (92)

  where the components $f_2$, $g_2$ are convex on V.
• Variable factor programming of the form:

  $$\underset{x,v}{min}\left[f(x,v)=\sum _{i=1}^q{v}_i{f}_i\left(x_i\right)\right]$$

  (93)

  s.t.

  $$g(x,v)=\sum _{i=1}^q{v}_i{x}_i-c\le 0$$

  (94)

  $$Av\le b,v\ge 0,v\in {\mathbb{R}}^q$$

  (95)

  $$x_i\ge 0,x_i,c\in {\mathbb{R}}^m$$

  (96)

  where $v_i,i=1,\dots ,q$ are non-negative scalars and $c,x_i,i=1,\dots ,q$ are vectors of the dimension m. A special case of this type of problem is the bilinear problem, which is nonconvex [24].

If the problem (4)–(7) is neither separable nor with variable factors, but convex with respect to the full vector of variables $(x,v)$, the cuts (90) can also be used.

Theorem 6.

For convex problems with respect to the full vector $(x,v)$ the evaluation (90) is valid.

Proof. 

For convex problems with differentiable functions (if the Lagrange function is not differentiable, the gradient can be replaced with a subgradient) we have for fixed $\lambda ={\lambda }^{*}$ and all x and v:

$$L_o(x,v,{\lambda }^{*})\ge L_o(x^{*},v^{*},{\lambda }^{*})+\frac{\partial L_o^T}{\partial x}(x^{*},v^{*},{\lambda }^{*})(x-x^{*})$$

$$+\frac{\partial L_o^T}{\partial v}(x^{*},v^{*},{\lambda }^{*})(v-v^{*})$$

(97)

This inequality will be preserved when we compute the minimum with respect to x on both sides:

$$\underset{x\in X}{min}{L}_o(x,v,{\lambda }^{*})\ge \underset{x\in X}{min}\left\{L_o(x^{*},v^{*},{\lambda }^{*})+\frac{\partial L_o^T}{\partial x}(x^{*},v^{*},{\lambda }^{*})(x-x^{*})+\frac{\partial L_o^T}{\partial v}(x^{*},v^{*},{\lambda }^{*})(v-v^{*})\right\}$$

$$=L_o(x^{*},v^{*},{\lambda }^{*})+\frac{\partial L_o^T}{\partial v}(x^{*},v^{*},{\lambda }^{*})(v-v^{*})+\underset{x\in X}{min}\frac{\partial L_o^T}{\partial x}(x^{*},v^{*},{\lambda }^{*})(x-x^{*})$$

(98)

Note that the last component of the expression (98) is equal to zero, as the point $x^{*}$ is optimal for $v=v^{*},\lambda ={\lambda }^{*}$ (there is no feasible direction of improvement, i.e., with a negative directional derivative), so the inequality (90) is valid. □

7. Computational Algorithm

We will now consider the case when at the optimal point of the primal problem $x^{*}$ in a version for both $v^{*}\in V\cap V_0$, that is (65)–(66), as well as $v^{*}\in V\backslash V_0$, that is (73)–(74), there exists a vector of Lagrange multipliers ${\lambda }^{*}$. Therefore, these problems must fulfill certain regularity conditions (more on that below).

A computational algorithm based on the generalized Benders method can be formulated as follows:

GBD Algorithm

• Choose a starting point $v^0\in V$; $K_o:=\varnothing ,K_f:=\varnothing ,k:=0$, take a convergence tolerance $\epsilon >$ 0.
• For the fixed $v=v^k$ try to solve the primal problem (65)–(66).
  • If successful$k:=k+1$, remember as $x^k$ and ${\lambda }^k$ the obtained optimal point and the optimal vector of Lagrange multipliers, $K_o:=K_o\cup \left\{k\right\}$. Update $\mathrm{UBD}=min\{\mathrm{UBD},z\left(v^k\right)\}$. If there has been an improvement in the estimate of the upper bound UBD, remember a pair $(x^k,v^k)$ as the best solution so far. If $\mathrm{UBD}-\mathrm{LBD}\le \epsilon$ then STOP.
  • Otherwise solve the feasibility problem (73)–(74) (as far as the used solver does not solve it itself facing infeasibility), $k:=k+1$, remember as $x^k$ and ${\lambda }^k$ the optimal point obtained and the vector of optimal Lagrange multipliers corresponding to constraints, $K_f:=K_f\cup \left\{k\right\}$.
• Solve the relaxed master problem:
  • in nonlinear version

    $$\underset{v\in V,\mu }{min}\mu$$

    (99)

    s.t.

    $$L_o(x^l,v,{\lambda }^l)\le \mu ,l\in K_o$$

    (100)

    $$L_f(x^l,v,{\lambda }^l)\le 0,l\in K_f$$

    (101)
  • or linearized

    $$\underset{v\in V,\mu }{min}\mu$$

    (102)

    s.t.

    $$L_o(x^l,v^l,{\lambda }^l)+\frac{\partial L_o^T}{\partial v}(x^l,v^l,{\lambda }^l)(v-v^l)\le \mu ,l\in K_o,$$

    (103)

    $$L_f(x^l,v^l,{\lambda }^l)+\frac{\partial L_f^T}{\partial v}(x^l,v^l,{\lambda }^l)(v-v^l)\le 0,l\in K_f$$

    (104)
  Let $(v^k,{\mu }^k)$ be the optimal solution of the above problem. Then ${\mu }^k$ is the lower estimate of the optimal objective of the original problem. Let $\mathrm{LBD}={\mu }^k$. If $\mathrm{UBD}-\mathrm{LBD}\le \epsilon$ then STOP.
• Go to 2.

Convergence of the Algorithm

It has been proved that the GBD algorithm converges in a finite number of steps for any $\epsilon >0$, when [2]:

• V is a finite discrete set, and the assumptions of Theorems 2 and 3 for Case 1 (then the same is also true for $\epsilon =$ 0) are satisfied.
• V is a nonempty, compact subset of $V_0$, X is a nonempty compact convex set, functions f and g are convex on X for each fixed $v\in V$ and continuous on $X\times V$, the set of optimal Lagrange multipliers in problem (65)–(66) is non-empty for all $v\in V_0$ and constraints satisfy Slater’s regularity condition: $\exists \overline{x}\in X,\overline{v}\in Vg(\overline{x},\overline{v})<0$.
  In the case when inequality constraints result from the application of the transformation (9) or (10) to equality constraints (8), Slater’s condition will be fulfilled, when you relax them using the epsilon tube method.
• V is not a subset of $V_0$ the constraint function $g(x,v)$ is separable and the set X is defined using linear constraints; other conditions are as in point 2 [4].

A very important assumption is that there exist Lagrange multipliers for all $v\in V\cap V_0$. If this is not checked, serious errors can appear. Consider the following example given (in the context of a slightly different approach to decomposition of optimization problems) by Grothey et al. [26]. For the sake of completeness a X set has been added, defined as a box being a Cartesian product of intervals $[-10.10]$ along every coordinate. It does not matter for the reasoning.

$$\underset{x_1,x_2,v}{min}{v}^2-x_2$$

(105)

s.t.

$${(x_1-1)}^2+x_2^2\le lnv$$

(106)

$${(x_1+1)}^2+x_2^2\le lnv$$

(107)

$$v\ge 1$$

(108)

$$-10\le x_i\le 10,i=1,2$$

(109)

The admissible area for the case when $v>e$ is presented in Figure 1.

For $v^{*}=e$ we will have the problem:

$$\underset{x_1,x_2,v}{min}{e}^2-x_2$$

(110)

s.t.

$${(x_1-1)}^2+x_2^2\le 1$$

(111)

$${(x_1+1)}^2+x_2^2\le 1$$

(112)

$$-10\le x_i\le 10,i=1,2$$

(113)

Its optimal solution is the only feasible point (0,0). It is easy to check that at this point the Lagrange multipliers for the constraints (111)–(112) do not exist, so the expressions (100) and (103) lose their sense. It results from the fact that the Fiacco–McCormick regularity condition (that gradients of active constraints are linearly independent) is not satisfied at this point. Interestingly, the primal problem (105)–(109), with respect to the full vector $(x,v)$, is convex and regular—this condition is met also at point $(x,v)=(0,0,e)$.

8. Effective Algorithms for Solving the Master Problem for Discrete Variables in the Version with Linear Cuts

Here we will discuss how to effectively solve the linear master problem (102)–(104) when $V\subseteq {\mathbb{Z}}^q$.

The GBD algorithm has some serious disadvantages. The most important of them is the number of constraints, which grows with every solution of the master problem, making it more complicated, and, hence, extending its maximum and average (in a certain window, because there can always be short single executions of the master problem, e.g., if the next optimal point $v^{*}$ is close to a given starting point, but this is not a typical thing) execution time. Moreover, there is no guarantee that in the next steps the value UBD of the approximation of the feasible solution will decrease. Therefore, it is worthwhile to use more sophisticated algorithms.

Nowadays the most useful as well as the most commonly used are the algorithms based on the cutting plane method, especially in conjunction with the interior point method [27].

The classical cutting plane method (more precisely the “cutting off plane method”) by Kelley [28], to which the master problem is reduced in the version with linearized cuts, has a number of disadvantages. First of all, it converges slowly. There are assessments which state that, to reach a solution with an accuracy of $\epsilon >0$, this method requires $O(\frac{1}{{\epsilon }^{q+1}})$ iterations [29]. A serious problem, mentioned above, is also that, as the calculations progress, the growing number of cuts causes subsequent iterations to be longer and longer. Unfortunately, despite many studies having been conducted, including recent machine learning techniques [8], there are no reliable and simple rules for removing old cuts, even those which are inactive for the current solution of the problem (102)–(104) [30]. One can also observe some kind of instability when the next point generated by the algorithm may be far from the previous one, even though the previous one was already close to the optimum, and even within it [25]. This effect may be alleviated by additional distance control constraints for each set of binary variables, limiting the integer update to a certain distance from the actual integer variable [31].

The above drawbacks have been largely removed through modifications of the Kelley method [27]. Unfortunately, from the most popular bundle method, which consists in adding a quadratic proximal component to a linear approximation of the objective function, which is a penalty for deviation from the last significant solution [32,33], very little can be expected in theory in the discrete case and “the mere production of a feasible point is never guaranteed” [34].

Much better, perhaps the best in this (i.e., MINLP) case, is the largest inscribed sphere method [35,36]. In this method the concept of a localization set is used. For the best estimate so far (say up to the k-th iteration of the master problem) of the upper optimal value of the objective function of the initial problem—let us denote it with $UB{D}^k$—it will be the set:

$$L_k=\left\{\begin{array}{ccc}\hfill (v,\mu )& \in & {\mathbb{Z}}^q\times \mathbb{R}\hfill \\ \hfill \mu & \le & UB{D}^k\hfill \\ \hfill \phi \left(v^l\right)+\nabla \phi {\left(v^l\right)}^T(v-v^l)& \le & \mu ,l\in K_o\hfill \\ \hfill \xi \left(v^l\right)+\nabla \xi {\left(v^l\right)}^T(v-v^l)& \le & 0,l\in K_f\hfill \end{array}\right\}$$

(114)

where:

$$\phi \left(v^l\right)=L_o(x^l,v^l,{\lambda }^l)=f(x^l,v^l)+{\left({\lambda }^l\right)}^{T}g(x^l,v^l)$$

(115)

$$\nabla \phi \left(v^l\right)=\frac{\partial L_o}{\partial v}(x^l,v^l,{\lambda }^l)=\frac{\partial f}{\partial v}(x^l,v^l)+\frac{\partial g}{\partial v}(x^l,v^l){\lambda }^l$$

(116)

$$\xi \left(v^l\right)=L_f(x^l,v^l,{\lambda }^l)={\left({\lambda }^l\right)}^{T}g(x^l,v^l)$$

(117)

$$\nabla \xi \left(v^l\right)=\frac{\partial L_f}{\partial v}(x^l,v^l,{\lambda }^l)=\frac{\partial g}{\partial v}(x^l,v^l){\lambda }^l$$

(118)

Note, that by assuming $u=(v,\mu )$, we can express the constraints that describe this set in the following way:

$$Au\le b$$

(119)

where matrix A is $S\times (q+1)$, with $S=({\overline{\overline{K}}}_o+{\overline{\overline{K}}}_f+1)$. The largest inscribed sphere method is one of the central point methods (from the larger family of interior point methods), which rely on looking in the subsequent steps k, instead of the point minimizing the objective function (102), for a point which, according to some measure, lies “in the middle” of the localization set $L_k$ (114).

The center of the largest inscribed sphere for the set $U=\{u\in {\mathbb{Z}}^q\times \mathbb{R}:Au\le b\}$ (called also the Chebyshev center) is determined by solving a linear programming problem:

$$\underset{u\in {\mathbb{Z}}^q\times \mathbb{R},\sigma \ge 0}{max}\sigma$$

(120)

s.t.

$$a_i^{T}u+\parallel a_i\parallel \sigma \le b_i,i=1,\dots ,S$$

(121)

The advantage of this method is, in addition to a linear formulation, the ease of eliminating unnecessary (distant) cuts, based on zero values of Lagrange multipliers (inactive constraints (121)), if in subsequent iterations of the master problem, the value of $\widehat{\sigma }$ decreased [35]. This method needs $O((q+1)ln\frac{1}{\epsilon })$ iterations to achieve the optimal solution with the accuracy of $\epsilon$ [29].

The next possibility is solving at the master level instead of (102)–(104) so-called outer approximation (OA) problem [37]:

$$\underset{x\in X,v\in V,\mu \in \mathbb{R}}{min}\mu$$

(122)

$$f(x^l,v^l)+\nabla f{(x^l,v^l)}^T\left[\begin{array}{c}x-x^l\\ v-v^l\end{array}\right]\le \mu ,l\in K_o$$

(123)

$$g(x^l,v^l)+\nabla g{(x^l,v^l)}^T\left[\begin{array}{c}x-x^l\\ v-v^l\end{array}\right]\le 0,l\in K_f$$

(124)

GBD can be regarded as a particular case of the OA method (because constraints in GBD are linear combinations of those in OA), and the lower bounds of GBD are weaker (that is not of a greater value) than those of OA. It means that OA used at the master lever requires a smaller number of iterations, however at a higher cost for solving the master problem, since the number of constraints added per iteration is greater [38].

Instead of calling many times at the master level, an external Mixed Integer Linear Programming (MILP) solver, which is time costly, more advanced users can write their own specialized solvers which avoid it. The simplest way is to use the branch-and-cut algorithm proposed in [39], which combines the above OA method and branch and bound algorithm. Cuts—both optimality and feasibility—are made when a relaxed (to continuous) master problem delivers a discrete solution $v^{*}\in V$, otherwise, in the case of a better than the discrete solution best so far, a partition of the domain is performed (branching) along a chosen variable $v_i^{*}$ with a fractional value or, in the opposite case, the given region (node) is fathomed [39]. Some improvements to this approach were proposed in [40].

In Boolean problems, that is where $V\subseteq {\{0,1\}}^q$, it may be useful to add the following cuts, assuring that the old solutions will not appear again [41]:

$$\sum _{j\in B^l}{v}_j-\sum _{j\in N^l}{v}_j\le \left|B^l\right|-1,l\in K_o\cup K_f$$

(125)

where $B^l=\{j|v_j^l=1\}$ and $N^l=\{j|v_j^l=0\}$.

Last but not least, it is always possible to extend the master level algorithm with some heuristics or machine learning techniques, connected more or less to specific features of the practical problems solved, e.g., [8,42,43,44,45,46,47,48].

The latest tests confirm that owing to the Benders method as a big problem as having 1956 binary and 57,616 continuous variables the three-scenario batch plant design problem can be solved with a 9.45% relative optimality gap in 729 s on a machine with 12 Xeon Intel processors, while a general solver BARON in 50,000 s delivers a solution with a 22.3% optimality gap (such solvers as SBB, Alpha-ECP, DICOPT failed to deliver a feasible solution in 50,000 s) [14].

9. Conclusions

The Benders method, although it was proposed sixty years ago, is still very vital. It is often used, especially in the version generalized by Geoffrion, to solve many large practical problems in the field of technology, mainly related to all types of networks, e.g., telecommunications, energy, gas, transport. These problems are characterized by mixed, discrete–continuous nature of decision variables, which is often accompanied by a nonconvexity of objective functions and constraints, but of such a type that, after fixing one of two subvectors of the problem variables, the problem becomes convex with respect to the second subvector (such are, for example, bilinear problems). Then the Benders method converges in a finite number of steps to exact solutions. This method copes well with the infeasibility of primal problems—with respect to the vector x—for some values of the complicating variables v, solving the feasibility problem from which new constraints are obtained—feasibility cuts. When the primal problem is feasible, the optimality cut is generated, eliminating part of the admissible set of complicating variables V, for which the objective function has worse values than those obtained so far.

There are two versions of the Benders method: with nonlinear and linear cuts. Both can be used when the condition of independence of the solution of the Lagrangian minimization problem in the primal problem from the vector of complicating variables is met as well as when the Lagrangian is convex with respect to complicating variables. Linear cuts can also be used in problems convex with respect to the full vector $(x,v)$.

The method with linear cuts (which in fact is a version of the Kelley cutting plane method) seems to be more practical, due to the ease of solving the master problem, e.g., using linear or quadratic solvers. It has many advantages, especially when using the largest inscribed sphere method (from the area of nondifferentiable optimization), outer approximation method or outer-approximation-based branch-and-cut method. In particular, practical problems, heuristics and machine learning techniques can additionally increase the efficiency of the master level algorithm.