Polyhedral DC Decomposition and DCA Optimization of Piecewise Linear Functions

Abstract

For piecewise linear functions $f:{\mathbb{R}}^n\mapsto \mathbb{R}$ we show how their abs-linear representation can be extended to yield simultaneously their decomposition into a convex $\check{f}$ and a concave part $\widehat{f}$, including a pair of generalized gradients $\check{g}\in {\mathbb{R}}^n\ni \widehat{g}$. The latter satisfy strict chain rules and can be computed in the reverse mode of algorithmic differentiation, at a small multiple of the cost of evaluating f itself. It is shown how $\check{f}$ and $\widehat{f}$ can be expressed as a single maximum and a single minimum of affine functions, respectively. The two subgradients $\check{g}$ and $-\widehat{g}$ are then used to drive DCA algorithms, where the (convex) inner problem can be solved in finitely many steps, e.g., by a Simplex variant or the true steepest descent method. Using a reflection technique to update the gradients of the concave part, one can ensure finite convergence to a local minimizer of f, provided the Linear Independence Kink Qualification holds. For piecewise smooth objectives the approach can be used as an inner method for successive piecewise linearization.

1. Introduction and Notation

There is a large class of functions $f:{\mathbb{R}}^n\mapsto \mathbb{R}$ that are called DC because they can be represented as the difference of two convex functions, see for example [1,2]. This property can be exploited in various ways, especially for (hopefully global) optimization. We find it notationally and conceptually more convenient to express these functions as averages of a convex and a concave function such that

$$f\left(x\right)=\frac{1}{2}(\check{f}\left(x\right)+\widehat{f}\left(x\right))\mathrm{with}\check{f}\left(x\right)\mathrm{convex}\mathrm{and}\widehat{f}\left(x\right)\mathrm{concave}.$$

Throughout we will annotate the convex part by superscript  $\stackrel{ˇ}{}$ and the concave part by superscript $\widehat{}$, which seems rather intuitive since they remind us of the absolute value function and its negative. Since we are mainly interested in piecewise linear functions we assume without much loss of generality that the functions f and the convex and concave components are well defined and finite on all of the Euclidean space ${\mathbb{R}}^n$. Allowing both components to be infinite outside their proper domain would obviously generate serious indeterminacies, i.e., $\mathtt{NaN}$s in the numerical sense. As we will see later we can in fact ensure in our setting that pointwise

$$\widehat{f}\left(x\right)\le f\left(x\right)\le \check{f}\left(x\right)\mathrm{for}\mathrm{all}x\in {\mathbb{R}}^n,$$

(1)

which means that we actually obtain an inclusion in the sense of interval mathematics [3]. This is one of the attractions of the averaging notation. We will therefore also refer to $\widehat{f}$ and $\check{f}$ as the concave and convex bounds of f.

Conditioning of the Decomposition

In parts of the literature the two convex functions $\check{f}$ and $-\widehat{f}$ are assumed to be nonnegative, which has some theoretical advantages. In particular, see, e.g., [4], one obtains for the square $h=f^2$ of a DC function f the decomposition

$$h=\frac{1}{4}{(\check{f}+\widehat{f})}^2=\frac{1}{2}\left\{\underset{\equiv \check{h}}{\underbrace{\frac{1}{4}({\check{f}}^2+{\widehat{f}}^2)}}+\underset{\equiv \widehat{h}}{\underbrace{\frac{1}{4}[-{(\check{f}-\widehat{f})}^2]}}\right\}.$$

(2)

The sign conditions of $\check{f}$ and $\widehat{f}$ are necessary to ensure that the three squares on the right hand side are convex functions. Using the Apollonius identity $f·h=\frac{1}{2}[{(f+h)}^2-f^2-h^2]$ one may then deduce in a constructive way that not only sums but also products of DC functions inherit this property. In general, since the convex functions $\check{f}$ and $-\widehat{f}$ have both supporting hyperplanes one can at least theoretically always find positive coefficients $\alpha$ and $\beta$ such that

$${\check{f}\left(x\right)+\alpha +\beta \parallel x\parallel }^2\ge 0\ge \widehat{f}\left(x\right)-\alpha -\beta {\parallel x\parallel }^2\mathrm{for}x\in {\mathbb{R}}^n.$$

Then the average of these modified functions is still f and their respective convexity/concavity properties are maintained. In fact, this kind of proximal shift can be used to show that any twice Lipschitz continuously differentiable function is DC, which raises the suspicion that the property by itself does not provide all that much exploitable structure from a numerical point of view. We believe that for its use in practical algorithms one has to make sure or simply assume that the condition number

$$\kappa (\check{f},\widehat{f})\equiv \underset{x\in {\mathbb{R}}^n}{sup}\frac{\left|\check{f}\left(x\right)\right|+|\widehat{f}\left(x\right)|}{|\check{f}\left(x\right)+\widehat{f}\left(x\right)|}\in [1,\infty ]$$

is not too large. Otherwise, there is the danger that the value of f is effectively lost in the rounding error of evaluating $\check{f}+\widehat{f}$. For sufficiently large quadratic shifts of the nature specified above one has $\kappa \sim \beta$. The danger of an excessive growth in $\kappa$ seems akin to the successive widening in interval calculations and similarly stems also from the lack of strict arithmetic rules. For example doubling f and then subtracting it yields the successive decompositions

$$\left(2f\right)-f=\left(\check{f}+\widehat{f}\right)-\frac{1}{2}\left(\check{f}+\widehat{f}\right)=(\check{f}-\frac{1}{2}\widehat{f})+(\widehat{f}-\frac{1}{2}\check{f})=\frac{1}{2}\left[(2\check{f}-\widehat{f})+(2\widehat{f}-\check{f})\right].$$

(3)

If in Equation (3) by chance we had originally $-\widehat{f}=\frac{1}{2}\check{f}>0$ so that $f=\frac{1}{2}\check{f}$ with the condition number $\kappa (\check{f},-0.5\check{f})=3$ we would get after the doubling and subtraction the condition number $\kappa (2.5\check{f},-2\check{f})=9$. So it is obviously important that the original algorithm avoids as much as possible calculations that are ill-conditioned in that they even just partly compensate each other.

Throughout the paper we assume that the functions in question are evaluated by a computational procedure that generates a sequence of intermediate scalars, which we denote generically by $u,v$ and w. The last one of these scalar variables is the dependent, which is usually denoted by f. All of them are continuous functions $u=u\left(x\right)$ of the vector $x\in {\mathbb{R}}^n$ of independent variables. As customary in mathematics we will often use the same symbol to identify a function and its dependent variable. For the overall objective we will sometimes distinguish them and write $y=f\left(x\right)$. For most of the paper we assume that the intermediates are obtained from each other by affine operations or the absolute value function so that the resulting $u\left(x\right)$ are all piecewise linear functions.

The paper is organized as follows. In the following Section 2 we develop rules for propagating the convex/concave decomposition through a sequence of abs-linear operations applied to intermediate quantities u. This can be done either directly on the pair of bounds $(\check{u},\widehat{u})$ or on their average u and their halved distance $\delta u=\frac{1}{2}(\check{u}-\widehat{u})$. In Section 3 we organize such sequences into an abs-linear form for f and then extend it to simultaneously yield the convex/concave decomposition. As a consequence of this analysis we get a strengthened version of the classical $max-min$ representation of piecewise linear functions, which reduces to the difference of two polyhedral parts in max- and min-form. In Section 4 we develop strict rules for propagating certain generalized gradient pairs $(\check{g},\widehat{g})$ of $(\check{u},\widehat{u})$ exploiting convexity and the cheap gradient principle [5]. In Section 5 we discuss the consequences for the DCA when using limiting gradients $(\check{g},\widehat{g})$, solving the inner, linear optimization problem (LOP) exactly, and ensuring optimality via polyhedral reflection. In Section 6 we demonstrate the new results on the nonconvex and piecewise linear chained Rosenbrock version of Nesterov [6]. Section 7 contains a summary and preliminary conclusion with outlook. In the Appendix A we give the details of the necessary and sufficient optimality test from [7] in the present DC context.

2. Propagating Bounds and/or Radii

In Equation (3) we already assumed that doubling is done componentwise and that for a difference $v=w-u$ of DC functions w and u, one defines the convex and concave parts by

$$\check{(w-u)}=\check{w}-\widehat{u}\mathrm{and}\widehat{(w-u)}=\widehat{w}-\check{u},$$

respectively. This yields in particular for the negation

$$\check{(-u)}=-\widehat{u}\mathrm{and}\widehat{(-u)}=-\check{u}.$$

(4)

For piecewise linear functions we need neither the square formula Equation (2) nor the more general decompositions for products. Therefore we will not insist on the sign conditions even though they would be also maintained automatically by Equation (4) as well as the natural linear rules for the convex and concave parts of the sum and the multiple of a DC function, namely

$$\begin{array}{cccccc}\hfill \check{(w+u)}& =(\check{w}+\check{u})\hfill & \mathrm{and}\hfill & \hfill \widehat{(w+u)}& =(\widehat{w}+\widehat{u}),\hfill & \hfill \\ \hfill \check{\left(cu\right)}& =c\check{u}\hfill & \mathrm{and}\hfill & \hfill \widehat{\left(cu\right)}& =c\widehat{u}\hfill & \mathrm{if}c\ge 0,\hfill \\ \hfill \check{\left(cu\right)}& =c\widehat{u}\hfill & \mathrm{and}\hfill & \hfill \widehat{\left(cu\right)}& =c\check{u}\hfill & \mathrm{if}c\le 0.\hfill \end{array}$$

However, the sign conditions would force one to decompose simple affine functions $u\left(x\right)=a^{\top }x+\beta$ as

$$u\left(x\right)=max(0,a^{\top }x+\beta )+min(0,a^{\top }x+\beta )\equiv \frac{1}{2}(\check{u}\left(x\right)+\widehat{u}\left(x\right)),$$

(5)

which does not seem such a good idea from a computational point of view.

The key observation for this paper is that as is well known (see e.g., [8]), one can propagate the absolute value operation according to the identity

$$\begin{array}{ccc}\hfill \left|u\right|& =& max(u,-u)=\frac{1}{2}max(\check{u}+\widehat{u},-\check{u}-\widehat{u})\hfill \\ & =& max(\check{u},-\widehat{u})+\frac{1}{2}(\widehat{u}-\check{u})\hfill \\ & \iff & \check{\left|u\right|}=2max(\check{u},-\widehat{u})\mathrm{and}\widehat{\left|u\right|}=\widehat{u}-\check{u}.\hfill \end{array}$$

(6)

Here the equality in the second line can be verified by shifting the difference $\frac{1}{2}(\widehat{u}-\check{u})$ into the two arguments of the max. Again we see that when applying the absolute value operation to an already positive convex function $u=\frac{1}{2}\check{u}\ge 0$ we get $\check{\left|u\right|}=2\check{u}$ and $\widehat{\left|u\right|}=-\check{u}$ so that the condition number grows from $\kappa (\check{u},0)=1$ to $\kappa (2\check{u},-\check{u})=3$. In other words, we observe once more the danger that both component functions drift apart. This looks a bit like simultaneous growth of numerator and denominator in rational arithmetic, which can sometimes be limited through cancelations by common integer factors. It is currently not clear when and how a similar compactification of a given convex/concave decomposition can be achieved. The corresponding rule for the maximum is similarly easy derived, namely

$$\begin{array}{cc}\hfill max(u,w)& =\frac{1}{2}max(\check{u}+\widehat{u},\check{w}+\widehat{w})=\frac{1}{2}\left(max(\check{u}-\widehat{w},\check{w}-\widehat{u})+\left(\widehat{u}+\widehat{w}\right)\right).\hfill \end{array}$$

When u and w as well as their decomposition are identical we arrive at the new decomposition $u=max(u,u)=\frac{1}{2}\left((\check{u}-\widehat{u})+2\widehat{u}\right)$, which obviously represents again some deterioration in the conditioning.

While it was pointed out in [4] that the DC functions $u=\frac{1}{2}(\check{u}+\widehat{u})$ themselves form an algebra, their decomposition pairs $(\check{u},\widehat{u})$ are not even an additive group, as only the zero $(0,0)$ has a negative partner, i.e., an additive inverse. Naturally, the pairs $(\check{u},\widehat{u})$ form the Cartesian product between the convex cone of convex functions and its negative, i.e., the cone of concave functions. The DC functions are then the linear envelope of the two cones in some suitable space of locally Lipschitz continuous functions. It is not clear whether this interpretation helps in some way, and in any case we are here mainly concerned with piecewise linear functions.

Propagating the Center and Radius

Rather than propagating the pairs $(\check{u},\widehat{u})$ through an evaluation procedure as defined in [5] to calculate the function value $f\left(x\right)$ at a given point x, it might be simpler and better for numerical stability to propagate the pair

$$u=\frac{1}{2}\left(\check{u}+\widehat{u}\right)\wedge \delta u=\frac{1}{2}(\check{u}-\widehat{u})\iff \check{u}=u+\delta u\wedge \widehat{u}=u-\delta u.$$

(7)

This representation resembles the so-called central form in interval arithmetic [9] and we will call therefore u the central value and $\delta u$ the radius. In other words, u is just the normal piecewise affine intermediate function and the $\delta u$ is a convex distance function to the hopefully close convex and concave part. Should the potential blow-up discussed above actually occur, this will only effect $\delta u$ but not the central value u itself. Moreover, at least theoretically one might decide to reduce $\delta u$ from time to time making sure of course that the corresponding $\check{u}$ and $\widehat{u}$ as defined in Equation (7) stay convex and concave, respectively. The condition number now satisfies the bound

$$\begin{array}{ccc}\hfill \kappa (u+\delta u,u-\delta u)& =& \underset{x}{sup}\frac{|u+\delta u|+|u-\delta u|}{2\left|u\right|}\hfill \\ & =& \underset{x}{sup}{\displaystyle \frac{1}{2}}\left\{|1+\frac{\delta u}{u}|+|1-\frac{\delta u}{u}|\right\}\le 1+\underset{x}{sup}\left|\frac{\delta u}{u}\right|.\hfill \end{array}$$

Recall here that all intermediate quantities $u=u\left(x\right)$ are functions of the independent variable vector $x\in {\mathbb{R}}^n$. Naturally, we will normally only evaluate the intermediate pairs u and $\delta u$ at a few iterates of whatever numerical calculation one performs involving f so that we can only sample the ratio

$$\rho u\left(x\right)\equiv \left|\delta u\right(x)/u(x\left)\right|$$

pointwise, where the denominator is hopefully nonzero. We will also refer to this ratio as the relative gap of the convex/concave decomposition at a certain evaluation point x. The arithmetic rules for propagating radii of the central form in central convex/concave arithmetic are quite simple.

Lemma 1 (Propagation rules for central form).

With$c,d,x\in \mathbb{R}$two constants and an independent variable we have

$$\begin{array}{ccc}v=c+dx\hfill & \Rightarrow \delta v=0\hfill & \Rightarrow \rho v=0ifv\ne 0\hfill \\ v=u\pm w\hfill & \Rightarrow \delta v=\delta u+\delta w\hfill & \Rightarrow \rho v\le \frac{\left|u\right|+\left|w\right|}{|u\pm w|}max(\rho u,\rho w)\hfill \\ v=cu\hfill & \Rightarrow \delta v=\left|c\right|\delta u\hfill & \Rightarrow \rho v=\rho uifc\ne 0\hfill \\ v=\left|u\right|\hfill & \Rightarrow \delta v=\left|u\right|+2\delta u\hfill & \Rightarrow \rho v\in [1,1+2\rho u].\hfill \end{array}$$

(8)

Proof. 

The last rule follows from Equation (6) by

$$\begin{array}{ccc}\hfill \delta \left(\right|u\left|\right)& =& \frac{1}{2}\left(\check{\left|u\right|}-\widehat{\left|u\right|}\right)=max(\check{u},-\widehat{u})-\frac{1}{2}(\widehat{u}-\check{u})\hfill \\ & =& max(\check{u}-\delta u,-\widehat{u}-\delta u)+2\delta u\hfill \\ & =& max(u,-u)+2\delta u=\left|u\right|+2\delta u.\hfill \end{array}$$

 ☐

The first equation in Equation (8) means that for all quantities u that are affine functions of the independent variables x the corresponding radius $\delta u$ is zero so that $\check{u}=u=\widehat{u}$ until we reach the first absolute value. Notice that $\delta v$ does indeed grow additively for the subtraction just like for the addition. By induction it follows from the rules above for an inner product that

$$\delta \left(\sum _{j=1}^m{c}_j{u}_j\right)=\sum _{j=1}^m\left|c_j\right|\delta u_j,$$

(9)

where the $c_j\in \mathbb{R}$ are assumed to be constants. As we can see from the bounds in Lemma 1 the relative gap can grow substantially whenever one performs an addition of values with opposite sign or applies the absolute value operation. In contrast to interval arithmetic on smooth functions one sees that the relative gap, though it may be zero or small initially immediately jumps above 1 when one hits the first absolute value operation. This is not really surprising since the best concave lower bound on $u\left(x\right)=\left|x\right|$ itself is $\widehat{u}\left(x\right)=0$ so that $\delta u=\left|x\right|$, $\check{u}\left(x\right)=2\left|x\right|$ and thus $\rho u\left(x\right)=1$ constantly. On the positive side one should notice that throughout we do not lose sight of the actual central values $u\left(x\right)$, which can be evaluated with full arithmetic precision. In any case we can think of neither $\rho$ nor $\kappa \le 1+\rho$ as small numbers, but we must be content if they do not actually explode too rapidly. Therefore they will be monitored throughout our numerical experiments.

Again we see that the computational effort is almost exactly doubled. The radii can be treated as additional variables that occur only in linear operations and stay nonnegative throughout. Notice that in contrast to the (nonlinear) interval case we do not loose any accuracy by propagating the central form. It follows immediately by induction from Lemma 1 that any function evaluated by a evaluation procedure that comprises a finite sequence of

• initializations to independent variables
• multiplications by constants
• additions or subtractions
• absolute value applications

is piecewise affine and continuous. We will call these operations and the resulting evaluation procedure abs-linear. It is also easy to see that the absolute values $|·|$ can be replaced by the maximum $max(·,·)$ or the minimum $min(·,·)$ or the positive part function $max(0,·)$ or any combination of them, since they can all be mutually expressed in terms of each other and some affine operations. Conversely, it follows from the min-max representation established in [10] (Proposition  2.2.2) that any piecewise affine function f can be evaluated by such an evaluation procedure. Consequently, by applying the formulas Equations (4)–(6) one can propagate at the same time the convex and concave components for all intermediate quantities. Alternatively, one can propagate the centered form according to the rules given in Lemma 1. These rules are also piecewise affine so that we have a finite procedure for simultaneously evaluating $\check{u}$ and $\widehat{u}$ or u and $\delta u$ as piecewise linear functions. The combined computation requires about 2–3 times as many arithmetic operations and twice as many memory accesses. Of course due to the interdependence of the two components it is not possible to evaluate just one of them without the other. As we will see the same is true for the generalized gradients to be discussed later in Section 4.

3. Forming and Extending the Abs-Linear Form

In practice all piecewise linear objectives can be evaluated by a sequence of abs-linear operations, possibly after min and max have been rewritten as

$$min(u,w)=\frac{1}{2}(u+w-|u-w\left|\right)\mathrm{and}max(u,w)=\frac{1}{2}(u+w+|u-w\left|\right).$$

(10)

Our only restriction is that the number s of intermediate scalar quantities, say $z_i$, is fixed, which is true for example in the $max-min$ representation. Then we can immediately cast the procedure in matrix-vector notation as follows:

Lemma 2 (Abs-Linear Form).

Any continuous piecewise affine function$f:x\in {\mathbb{R}}^n\mapsto y\in \mathbb{R}$can be represented by

$$\begin{array}{cc}\hfill z& =c+Zx+Mz+L\left|z\right|,\hfill \\ \hfill y& =d+a^{\top }x+b^{\top }z,\hfill \end{array}$$

(11)

where$z\in {\mathbb{R}}^s,Z\in {\mathbb{R}}^{s\times n},M,L\in {\mathbb{R}}^{s\times s}$strictly lower triangular,$d\in \mathbb{R},a\in {\mathbb{R}}^n,b\in {\mathbb{R}}^s$and$\left|z\right|$denotes the componentwise modulus of the vector z.

It should be noted that the construction of this general abs-linear form requires no analysis or computation whatsoever. However, especially for our purpose of generating a reasonably tight DC decomposition, it is advantages to reduce the size of the abs-normal form by eliminating all intermediates $z_j$ with $j<s$ for which $|z_j|$ never occurs on the right hand side. To this end we may simply substitute the expression of $z_j$ given in the j-th row in all places where $z_j$ itself occurs on the right hand side. The result is what we will call a reduced abs-normal form, where after renumbering, all remaining $z_j$ with $j<s$ are switching variables in that $|z_j|$ occurs somewhere on the right hand side. In other words, all but the last column of the reduced, strictly lower triangular matrix L are nontrivial. Again, this reduction process is completely mechanical and does not require any nontrivial analysis, other than looking up which columns of the original L were zero. The resulting reduced system is smaller and probably denser, which might increase the computation effort for evaluating f itself. However, in view of Equation (9) we must expect that for the reduced form the radii will grow slower if we first accumulate linear coefficients and then take their absolute values. Hence we will assume in the remainder of this paper that the abs-normal form for our objective f of interest is reduced.

Based on the concept of abs-linearization introduced in [11], a slightly different version of a (reduced) abs-normal form was already proposed in [12]. Now in the present paper, both z and y depend directly on z via the matrix M and the vector b, but y does no longer depend directly on $\left|z\right|$. All forms can be easily transformed into each other by elementary modifications. The intermediate variables $z_i$ can be calculated successively for $1\le i\le s$ by

$$\begin{array}{c}\hfill z_i=c_i+Z_{i}x+M_{i}z+L_i\left|z\right|,\end{array}$$

(12)

where $Z_i$, $M_i$ and $L_i$ denote the ith rows of the corresponding matrix. By induction on i one sees immediately that they are piecewise affine functions $z_i=z_i\left(x\right)$, and we may define for each x the signature vector

$$\sigma \left(x\right)={(sgn\left(z_i\left(x\right)\right))}_{i=1\dots s}\in {\{-1,0,1\}}^s.$$

Consequently we get the inverse images

$${\mathcal{P}}_{\sigma }\equiv \{x\in {\mathbb{R}}^n:sgn\left(z\left(x\right)\right)=\sigma \}\mathrm{for}\sigma \in {\{-1,0,1\}}^s,$$

(13)

which are relatively open polyhedra that form collectively a disjoint decomposition of ${\mathbb{R}}^n$. The situation for the second example of Nesterov is depicted in Figure 3 in the penultimate section. There are six polyhedra of full dimension, seven polyhedra of co dimension 1 drawn in blue and two points, which are polyhedra of dimension 0. The point $(0,-1)$ with signature $(0,-1,0)$ is stationary and the point $(1,1)$ with signature $(1,0,0)$ is the minimizer as shown in [7]. The arrows indicate the path of our reflection version of the DCA method as described in Section 5.

When $\sigma$ is definite, i.e., has no zero components, which we will denote by $0\notin \sigma$, it follows from the continuity of $z\left(x\right)$ that ${\mathcal{P}}_{\sigma }$ has full dimension n unless it is empty. In degenerate situations this may also be true for indefinite $\sigma$ but then the closure of ${\mathcal{P}}_{\sigma }$ is equal to the extended closure

$${\stackrel{-}{\mathcal{P}}}_{\tilde{\sigma }}\equiv \{x\in {\mathbb{R}}^n:\sigma \left(x\right)\prec \tilde{\sigma }\}\supset close\left({\mathcal{P}}_{\tilde{\sigma }}\right)$$

(14)

for some definite $0\notin \tilde{\sigma }\succ \sigma$. Here the (reflexive) partial ordering ≺ between the signature vectors satisfies the equivalence

$$\mathring{\sigma }\prec \sigma \iff {\mathring{\sigma }}_i{\sigma }_i\le {\sigma }_i^2\mathrm{for}i=1\dots s\iff {\stackrel{-}{\mathcal{P}}}_{\mathring{\sigma }}\subset {\stackrel{-}{\mathcal{P}}}_{\sigma }$$

as shown in [13]. One can easily check that for any $\sigma \succ \mathring{\sigma }$ there exists a unique signature

$${(\sigma ▹\mathring{\sigma })}_i=\left\{\begin{array}{cc}{\sigma }_i\hfill & \mathrm{if}{\mathring{\sigma }}_i\ne 0\hfill \\ -{\sigma }_i\hfill & \mathrm{if}{\mathring{\sigma }}_i=0\hfill \end{array}\right.\mathrm{for}i=1\dots s$$

(15)

We call $\tilde{\sigma }\equiv \sigma ▹\mathring{\sigma }$ the reflection of $\sigma$ at $\mathring{\sigma }$, which satisfies also $\tilde{\sigma }\succ \mathring{\sigma }$ and we have in fact ${\stackrel{-}{\mathcal{P}}}_{\tilde{\sigma }}\cap {\stackrel{-}{\mathcal{P}}}_{\sigma }={\stackrel{-}{\mathcal{P}}}_{\mathring{\sigma }}$. Hence the relation between $\sigma$ and $\tilde{\sigma }$ is symmetric in that also $\sigma =\tilde{\sigma }▹\mathring{\sigma }$. Therefore we will call $(\sigma ,\tilde{\sigma })$ a complementary pair with respect to $\mathring{\sigma }$. In the very special case $z_i=x_i$ for $i=1\dots n=s-1$ the ${\stackrel{-}{\mathcal{P}}}_{\sigma }$ are orthants and their reflections at the origin $\left\{0\right\}={\stackrel{-}{\mathcal{P}}}_0\subset {\mathbb{R}}^n$ are their geometric opposites ${\stackrel{-}{\mathcal{P}}}_{\tilde{\sigma }}$ with $\tilde{\sigma }=-\sigma$. Here one can see immediately that all edges, i.e., one-dimensional polyhedra, have Cartesian signatures $\pm e_i$ for $i=1\dots n$ and belong to ${\stackrel{-}{\mathcal{P}}}_{\sigma }$ or ${\stackrel{-}{\mathcal{P}}}_{\tilde{\sigma }}$ for any given $\sigma$. Notice that $\mathring{x}$ is a local minimizer of a piecewise linear function if and and only if it is a local minimizer along all edges of nonsmoothness emanating form it. Consequently, optimality of f restricted to a complementary pair is equivalent to local optimality on ${\mathbb{R}}^n$, not only in this special case, but whenever the Linear Independence Kink Qualification (LIKQ) holds as introduced in [13] and defined in the Appendix A. This observation is the basis of the implicit optimality condition verified by our DCA variant Algorithm 1 through the use of reflections. The situation is depicted in Figure 3 where the signatures $(-1,-1,-1)$ and $(1,-1,1)$ as well as $(1,-1,1)$ and $(1,1,-1)$ form complementary pairs at $(0,-1)$ and $(1,1)$, respectively. At both reflection points there are four emanating edges, which all belong to one of the three polyhedra mentioned.

Applying the propagation rules from Lemma 1, one obtains with $\delta x=0\in {\mathbb{R}}^n$ the recursion

$$\begin{array}{cc}\hfill \delta z_1& =\delta (c_1+Z_{1}x)=0\hfill \\ \hfill \delta z_i& =\left(\right|M_i|+2|L_i\left|\right)\delta z+|L_i\left|\right|z|\mathrm{for}i=2\dots s,\hfill \end{array}$$

where the modulus is once more applied componentwise for vectors and matrices. Hence, we have again in matrix vector notation

$$\begin{array}{cc}\hfill \delta z& =\left(\right|M|+2|L\left|\right)\delta z+\left|L\right|\left|z\right|,\hfill \end{array}$$

(16)

which yields for $\delta z$ the explicit expression

$$\delta z={(I-|M|-2|L\left|\right)}^{-1}\left|L\right|\left|z\right|=\sum _{j=0}^{\nu }{\left(\right|M\left|+2\right|L\left|\right)}^j\left|L\right|\left|z\right|\ge 0.$$

(17)

Here, $\nu$ is the so-called switching depth of the abs-linear form of f, namely the largest $\nu \in \mathbb{N}$ such that ${\left(\right|M\left|+\right|L\left|\right)}^{\nu }\ne 0$, which is always less than s due to the strict lower triangularity of M and L. The unit lower triangular $(I-|M|-2|L\left|\right)$ is an M-matrix [14], and interestingly enough does not even depend on x but directly maps $\left|z\right|=\left|z\right(x\left)\right|$ to $\delta z=\delta z\left(x\right)$. For the radius of the function itself, the propagation rules from Lemma 1 then yield

$$\begin{array}{cc}\hfill \delta f\left(x\right)=\delta y& ={\left|b\right|}^{\top }\delta z\ge 0.\hfill \end{array}$$

(18)

This nonnegativity implies the inclusion Equation (1) already mentioned in Section 1, i.e.:

Theorem 1 (Inclusion by convex/concave decomposition).

For any piecewise affine function f in abs-linear form, the construction defined in Section 2 yields a convex/concave inclusion

$$\begin{array}{c}\hfill \widehat{f}\left(x\right)\le f\left(x\right)\equiv \frac{1}{2}(\check{f}\left(x\right)+\widehat{f}\left(x\right))\le \check{f}\left(x\right).\end{array}$$

Moreover, the convex and the concave parts$\check{f}\left(x\right)$and$\widehat{f}\left(x\right)$have exactly the same switching structure as$f\left(x\right)$in that they are affine on the same polyhedra${\mathcal{P}}_{\sigma }$defined in (13).

Proof. 

Equations (16) and (17) ensure that $\delta f\left(x\right)$ is nonnegative at all $x\in {\mathbb{R}}^n$ such that

$$\begin{array}{c}\hfill \widehat{f}\left(x\right)=f\left(x\right)-\delta f\left(x\right)\le f\left(x\right)\le f\left(x\right)+\delta f\left(x\right)\le \check{f}\left(x\right).\end{array}$$

It follows from Equation (17) that the radii $\delta z_i\left(x\right)$ are like the $|z_i\left(x\right)|$ piecewise linear with the only nonsmoothness arising through the switching variables $z\left(x\right)$ themselves. Obviously this property is inherited by $\delta f\left(x\right)$ and the linear combinations $\check{f}\left(x\right)=f\left(x\right)+\delta f\left(x\right)$ and $\widehat{f}\left(x\right)=f\left(x\right)-\delta f\left(x\right)$, which completes the proof. ☐

Combining Equations (16) and (18) with the abs-linear form of the piecewise affine function f and defining $\tilde{z}=(z,\delta z)\in {\mathbb{R}}^{2s}$, one obtains for the calculation of $\tilde{f}\left(x\right)\equiv \tilde{y}\equiv (y,\delta y)$ the following abs-linear form

$$\begin{array}{cc}\hfill \tilde{z}& =\tilde{c}+\tilde{Z}x+\tilde{M}\tilde{z}+\tilde{L}\left|\tilde{z}\right|,\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill \tilde{y}& =\tilde{d}+{\tilde{a}}^{\top }x+{\tilde{b}}^{\top }\tilde{z}\hfill \end{array}$$

(20)

with the vectors and matrices defined by

$$\begin{array}{cc}\hfill \tilde{c}& =\left[\begin{array}{c}c\\ 0\end{array}\right]\in {\mathbb{R}}^{2s},\tilde{Z}=\left[\begin{array}{c}Z\\ 0\end{array}\right]\in {\mathbb{R}}^{2s\times n},\tilde{M}=\left[\begin{array}{cc}M& 0\\ 0& \left|M\right|+2\left|L\right|\end{array}\right]\in {\mathbb{R}}^{2s\times 2s},\hfill \\ \hfill \tilde{L}& =\left[\begin{array}{cc}L& 0\\ \left|L\right|& 0\end{array}\right]\in {\mathbb{R}}^{2s\times 2s},\tilde{d}=\left[\begin{array}{c}d\\ 0\end{array}\right]\in {\mathbb{R}}^2,\tilde{a}=\left[\begin{array}{cc}a& 0\end{array}\right]\in {\mathbb{R}}^{n\times 2},\tilde{b}=\left[\begin{array}{cc}b& 0\\ 0& \left|b\right|\end{array}\right]\in {\mathbb{R}}^{2s\times 2}.\hfill \end{array}$$

Then, Equations (19) and (20) yield

$$\begin{array}{cc}\hfill \left[\begin{array}{c}z\\ \delta z\end{array}\right]& =\left[\begin{array}{c}c\\ 0\end{array}\right]+\left[\begin{array}{c}Z\\ 0\end{array}\right]x+\left[\begin{array}{cc}M& 0\\ 0& \left|M\right|+2\left|L\right|\end{array}\right]\left[\begin{array}{c}z\\ \delta z\end{array}\right]+\left[\begin{array}{cc}L& 0\\ \left|L\right|& 0\end{array}\right]\left[\begin{array}{c}\left|z\right|\\ \left|\delta z\right|\end{array}\right]=\left[\begin{array}{c}c+Zx+Mz+L\left|z\right|\\ \left(\right|M|+2|L\left|\right)\delta z+\left|L\right|\left|z\right|\end{array}\right]\hfill \\ \hfill \left[\begin{array}{c}y\\ \delta y\end{array}\right]& =\tilde{d}+{\tilde{a}}^{\top }x+{\tilde{b}}^{\top }\tilde{z}=\left[\begin{array}{c}d\\ 0\end{array}\right]+\left[\begin{array}{c}{a}^{\top }x\\ 0\end{array}\right]+{\left[\begin{array}{cc}b& 0\\ 0& \left|b\right|\end{array}\right]}^{\top }\left[\begin{array}{c}z\\ \delta z\end{array}\right]=\left[\begin{array}{c}d+a^{\top }x+b^{\top }z\\ {\left|b\right|}^{\top }\delta z\end{array}\right],\hfill \end{array}$$

i.e., Equations (16) and (18). As can be seen, the matrices $\tilde{M}$ and $\tilde{L}$ have the required strictly lower triangular form. Furthermore, it is easy to check, that the switching depth of the abs-linear form of f carries over to the abs-linear form for $\tilde{f}$ in that also $\left(\right|\tilde{M}|+|\tilde{L}{\left|\right)}^{\nu }\ne 0=\left(\right|\tilde{M}|+|\tilde{L}{\left|\right)}^{\nu +1}$. However, notice that this system is not reduced since the s radii are not switching variables, but globally nonnegative anyhow. We can now obtain explicit expressions for the central values, radii, and bounds for a given signature $\sigma$.

Corollary 1 (Explicit representation of the centered form).

For any definite signature$\sigma 0$and all$x\in {\mathcal{P}}_{\sigma }$we have with$\mathsf{\Sigma }=diag\left(\sigma \right)$

$$\begin{array}{cc}\hfill z_{\sigma }\left(x\right)& ={(I-M-L\mathsf{\Sigma })}^{-1}\left(c+Zx\right)and\left|z_{\sigma }\left(x\right)\right|=\mathsf{\Sigma }{z}_{\sigma }\left(x\right)\ge 0\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill \delta z_{\sigma }\left(x\right)& ={(I-|M|-2|L\left|\right)}^{-1}\left|L\right|\mathsf{\Sigma }{(I-M-L\mathsf{\Sigma })}^{-1}\left(c+Zx\right)\ge 0\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill \nabla z_{\sigma }& ={(I-M-L\mathsf{\Sigma })}^{-1}Z\Rightarrow {\nabla }_{\sigma }f=a^{\top }+b^{\top }{(I-M-L\mathsf{\Sigma })}^{-1}Z\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill \nabla f_{\sigma }& =a^{\top }+\left[b^{\top }{+\left|b\right|}^{\top }{(I-|M|-2|L\left|\right)}^{-1}\left|L\right|\mathsf{\Sigma }\right]{(I-M-L\mathsf{\Sigma })}^{-1}Z\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill \nabla {\widehat{f}}_{\sigma }& =a^{\top }+\left[b^{\top }-{\left|b\right|}^{\top }{(I-|M|-2|L\left|\right)}^{-1}\left|L\right|\mathsf{\Sigma }\right]{(I-M-L\mathsf{\Sigma })}^{-1}Z,\hfill \end{array}$$

(25)

where the restrictions of the functions and their gradients to${\mathcal{P}}_{\sigma }$are denoted by subscript σ. Notice that the gradients are constant on these open polyhedra.

Proof. 

Equations (21) and (23) follow directly from Equation (12), the abs-linear form (11) and the properties of $\mathsf{\Sigma }$. Combining Equation (16) with  (21) yields Equation (22). Since $\check{f}\left(x\right)=f\left(x\right)+\delta f\left(x\right)$ and $\widehat{f}\left(x\right)=f\left(x\right)-\delta f\left(x\right)$, Equations (24) and (25) follow from the representation in abs-linear form and Equation (23). ☐

As one can see the computation of the gradient $\nabla f_{\sigma }$ requires the solution of one unit upper triangular linear system and that of both $\nabla {\check{f}}_{\sigma }$ and $\nabla {\widehat{f}}_{\sigma }$ one more. Naturally, upper triangular systems are solved by back substitution, which corresponds to the reverse mode of algorithmic differentiation as described in the following section. Hence, the complexity for calculating the gradients is exactly the same as that for calculating the functions, which can be obtained by one forward substitution for $f_{\sigma }$ and an extra one for $\delta f_{\sigma }$ and thus ${\check{f}}_{\sigma }$ and ${\widehat{f}}_{\sigma }$. The given $\nabla f_{\sigma },\nabla {\check{f}}_{\sigma }$ and $\nabla {\widehat{f}}_{\sigma }$ are proper gradients in the interior of the full dimensional domains ${\mathcal{P}}_{\sigma }$. For some or even many $\sigma$ the inverse image ${\mathcal{P}}_{\sigma }$ of the map $x\mapsto sgn\left(z\right(x\left)\right)$ may be empty, in which case the formulas in the corollary do not apply. Checking the nonemptiness of ${\mathcal{P}}_{\sigma }$ for a given signature $\sigma$ amounts to checking the consistency of a set of linear inequalities, which costs the same as solving an LOP and is thus nontrivial. Expressions for the generalized gradients at points in lower dimensional polyhedra are given in the following Section 4. There it is also not required that the abs-linear normal form has been reduced, but one may consider any given sequence of abs-linear operations.

The Two-Term Polyhedral Decomposition

It is well known ([15], Theorem 2.49) that all piecewise linear and globally convex or concave functions can be represented as the maximum or the minimum of a finite collection of affine functions, respectively. Hence, from the convex/concave decomposition we get the following drastic simplification of the classical min-max representation given, e.g., in [10].

Corollary 2 (Additive max/min decomposition of PL functions).

For every piecewise affine function$f:{\mathbb{R}}^n\mapsto \mathbb{R}$there exist$k\ge 0$affine functions${\alpha }_i+a_i^{\top }x$for$i=1\dots k$and$l\ge 0$affine functions${\beta }_j+b_j^{\top }x$ for $j=1\dots l$ such that at all $x\in {\mathbb{R}}^n$

$$f\left(x\right)=\underset{\equiv \frac{1}{2}\check{f}\left(x\right)}{\underbrace{\underset{i=1\dots k}{max}({\alpha }_i+a_i^{\top }x)}}+\underset{\equiv \frac{1}{2}\widehat{f}\left(x\right)}{\underbrace{\underset{j=1\dots l}{min}({\beta }_j+b_j^{\top }x)}}$$

(26)

where furthermore $\widehat{f}\left(x\right)\le f\left(x\right)\le \check{f}\left(x\right)$.

The max-part of this representation is what is called a polyhedral function in the literature [15]. Since the min-part is correspondingly the negative of a polyhedral function we may also refer to Equation (26) as a DP decomposition, i.e., the difference of two polyhedral functions.

We are not aware of a publication that gives a practical procedure for computing such a collection of affine functions ${\alpha }_i+a_i^{\top }x$, $i=1\dots k$, and ${\beta }_j+b_j^{\top }x$, $j=1\dots l$, for a given piecewise linear function f. Of course the critical question is in which form the function f is specified. Here as throughout our work we assume that it is given by a sequence of abs-linear operations. Then we can quite easily compute for each intermediate variable v representations of the form

$$\begin{array}{ccc}\hfill v& =& \sum _{i=1}^{\overline{m}}\underset{1\le j\le k_i}{max}({\alpha }_{ij}+a_{ij}^{\top }x)+\sum _{i=1}^{\overline{n}}\underset{1\le j\le l_i}{min}({\beta }_{ij}+b_{ij}^{\top }x)\hfill \end{array}$$

(27)

$$\begin{array}{ccc}\hfill & =& \underset{\genfrac{}{}{0.0pt}{}{j_i\in I_i}{1\le i\le \overline{m}}}{max}\sum _{i=1}^{\overline{m}}({\alpha }_{i{j}_i}+a_{i{j}_i}^{\top }x)+\underset{\genfrac{}{}{0.0pt}{}{j_i\in J_i}{1\le i\le \overline{n}}}{min}\sum _{i=1}^{\overline{n}}({\beta }_{i{j}_i}+b_{i{j}_i}^{\top }x).\hfill \end{array}$$

(28)

with index sets $I_i=\{1,\dots ,k_i\},1\le i\le \overline{m},$ and $J_i=\{1,\dots ,l_i\},1\le i\le \overline{n},$ since one has to consider all possibilities of selecting one affine function each from one of the $\overline{m}$ max and $\overline{n}$ min groups, respectively. Obviously, (28) involves ${\prod }_{i=1}^m{k}_i$ and ${\prod }_{i=1}^n{\ell }_i$ affine function terms in contrast to the first representation (27) which contains just ${\sum }_{i=1}^m{k}_i$ and ${\sum }_{i=1}^n{\ell }_i$ of them. Still the second version conforms to the classical representation of convex and concave piecewise linear functions, which yields the following result:

Corollary 3 (Explicit computation of the DP representation).

For any piecewise linear function given as abs-linear procedure one can explicitly compute the representation (26) by implementing the rules of Lemma 1.

Proof. 

We will consider the representations (27) from which (26) can be directly obtained in the form (28). Firstly, the independent variables $x_j$ are linear functions of themselves with gradient $a=e_j$ and inhomogeneity $\alpha =0$. Then for multiplications by a constant $c>0$ we have to scale all affine functions by c. Secondly, addition requires appending the expansions of the two summands to each other without any computation. Taking the negative requires switching the sign of all affine functions and interchanging the max and min group. Finally, to propagate through the absolute values we have to apply the rule (6), which means switching the signs in the min group, expressing it in terms of max and merging it with the existing max group. Here merging means pairwise joining each polyhedral term of the old max-group with each term in the switched min-group. Then the new min-group is the old one plus the old max-group with its sign switched. ☐

We see that taking the absolute value or, alternatively, maxima or minima generates the strongest growth in the number of polyhedral terms and their size. It seems clear that this representation is generally not very useful because the number of terms will likely blow up exponentially. This is not surprising because we will need one affine function for each element of the polyhedral decompositions of the domain of the max and min term. Typically, many of the affine terms will be redundant, i.e., could be removed without changing the values of the polyhedral terms. Unfortunately, identifying those already requires solving primal or dual linear programming problems, see, e.g., [16]. It seems highly doubtful that this would ever be worthwhile. Therefore, we will continue to advocate dealing with piecewise linear functions in a convenient procedural abs-linear representation.

4. Computation of Generalized Gradients and Constructive Oracle Paradigm

For optimization by variants of the DCA algorithm [17] one needs generalized gradients of the convex and the concave component. Normally, there are no strict rules for propagating generalized gradients through nonsmooth evaluation procedures. However, exactly this is simply assumed in the frequently invoked oracle paradigm, which states that at any point $x\in {\mathbb{R}}^n$ the function value $f\left(x\right)$ and an element $g\in \partial f\left(x\right)$ can be evaluated. We have argued in [18] that this is not at all a reasonable assumption.

On the other hand, it is well understood that for the convex operations: Positive scaling, addition, and taking the maximum the rules are strict and simple. Moreover, then the generalized gradient in the sense of Clarke $\partial \check{f}\left(x\right)\subset {\mathbb{R}}^n$ is actually a subdifferential in that all its elements define supporting hyperplanes. Similarly $\partial \widehat{f}\left(x\right)$ might be called a superdifferential in that the tangent planes bound the concave part from above.

In other words, we have at all $x\in {\mathbb{R}}^n$ and for all increments $\Delta x$

$$\check{f}(x+\Delta x)\ge \check{f}\left(x\right)+{\check{g}}^{\top }\Delta x\mathrm{if}\check{g}\in \partial \check{f}\left(x\right)$$

and

$$\widehat{f}(x+\Delta x)\le \widehat{f}\left(x\right)+{\widehat{g}}^{\top }\Delta x\mathrm{if}\widehat{g}\in \partial \widehat{f}\left(x\right),$$

which imply for $\check{g}\in \partial \check{f}\left(x\right)$ and $\widehat{g}\in \partial \widehat{f}\left(x\right)$ that

$$\widehat{f}(x+\Delta x)+\check{f}\left(x\right)+{\check{g}}^{\top }\Delta x\le 2f(x+\Delta x)\le \check{f}(x+\Delta x)+\widehat{f}\left(x\right)+{\widehat{g}}^{\top }\Delta x,$$

(29)

where the lower bound on the left is a concave function and the upper bound is convex, both with respect to $\Delta x$. Notice that the generalized superdifferential $\partial \widehat{f}$ being the negative of the subdifferential of $-\widehat{f}$ is also a convex set.

Now the key question is how we can calculate a suitable pair of generalized gradients $(\check{g},\widehat{g})\in \partial \check{f}\left(x\right)\times \partial \widehat{f}\left(x\right)$. As we noted above the convex part and the negative of the concave part only undergo convex operations so that for $v=cu$

$$\partial \check{v}=\left\{\begin{array}{cc}c\partial \check{u}\hfill & \mathrm{if}c>0\hfill \\ 0\hfill & \mathrm{if}c=0\hfill \\ c\partial \widehat{u}\hfill & \mathrm{if}c<0\hfill \end{array}\right.\mathrm{and}\partial \widehat{v}=\left\{\begin{array}{cc}c\partial \widehat{u}\hfill & \mathrm{if}c>0\hfill \\ 0\hfill & \mathrm{if}c=0\hfill \\ c\partial \check{u}\hfill & \mathrm{if}c<0\hfill \end{array}\right.$$

(30)

and for $v=u+w$

$$\partial \check{v}=\partial \check{u}+\partial \check{w}\mathrm{and}\partial \widehat{v}=\partial \widehat{u}+\partial \widehat{w}.$$

(31)

Finally, for $v=\left|u\right|$ we find by Equation (6) that $\partial \widehat{v}=\partial \widehat{u}-\partial \check{u}$ as well as

$$\frac{1}{2}\partial \check{v}=\partial max(\check{u},-\widehat{u})=\left\{\begin{array}{cc}\partial \check{u}\hfill & \mathrm{if}u>0\hfill \\ conv\left\{\partial \check{u}\cup (-\partial \widehat{u})\right\}\hfill & \mathrm{if}u=0,\hfill \\ -\partial \widehat{u}\hfill & \mathrm{if}u<0\hfill \end{array}\right.$$

(32)

where we have used that $u=\frac{1}{2}(\check{u}+\widehat{u})$ in Equation (32). The sign of the arguments u of the absolute value function are of great importance, because they determine the switching structure. For this reason, we formulated the cases in terms of u rather than in the convex/concave components. The operator $conv\{·\}$ denotes taking the convex hull or envelope of a given usually closed set. It is important to state that within an abs-linear representation the multipliers c will stay constant independent of the argument x, even if they were originally computed as partial derivatives by an abs-linearization process and thus subject to round-off error. In particular their sign will remain fixed throughout whatever algorithmic calculation we perform involving the piecewise linear function f. So, actually the case $c=0$ could be eliminated by dropping this term completely and just initializing the left hand side v to zero.

Because we have set identities we can propagate generalized gradient pairs $(\nabla \check{u},\nabla \widehat{u})\in \partial \check{u}\times \partial \widehat{u}$ and perform the indicated algebraic operations on them, starting with the Cartesian basis vectors

$$\nabla {\check{x}}_j=\nabla {\widehat{x}}_j=\nabla x_j=e_j\mathrm{since}{\check{x}}_j={\widehat{x}}_j=x_j\mathrm{for}j=1\dots n.$$

The result of this propagation is guaranteed to be an element of $\partial \check{f}\times \partial \widehat{f}$. Recall that in the merely Lipschitz continuous case generalized gradients cannot be propagated with certainty since for example the difference $v=w-u$ generates a proper inclusion $\partial v\subset \partial w-\partial u$. In that vein we must emphasize that the average $\frac{1}{2}(\nabla \check{f}+\nabla \widehat{f})$ need not be a generalized gradient of $f=\frac{1}{2}\left(\check{f}+\widehat{f}\right)$ as demonstrated by the possibility that $\widehat{f}=-\check{f}$ algebraically but we happen to calculate different generalized gradients of $\check{f}$ and $-\widehat{f}$ at a particular point x. In fact, if one could show that $\partial f=\frac{1}{2}(\partial \check{f}+\partial \widehat{f})$ one would have verified the oracle paradigm, whose use we consider unjustified in practice. Instead, we can formulate another corollary for sufficiently piecewise smooth functions.

Definition 1.

For any$d\in \mathbb{N}$, the set of functions$f:{\mathbb{R}}^n\mapsto \mathbb{R},y=f\left(x\right),$defined by an abs-normal form

$$\begin{array}{ccc}\hfill z& =& F(x,z,|z\left|\right),\hfill \\ \hfill y& =& \phi (x,z),\hfill \end{array}$$

with$F\in {\mathcal{C}}^d\left({\mathbb{R}}^{n+s+s}\right)$and$\phi \in {\mathcal{C}}^d\left({\mathbb{R}}^{n+s}\right)$, is denoted by${\mathcal{C}}_{\mathrm{abs}}^d\left({\mathbb{R}}^n\right).$

Once more, this definition differs slightly from the one given in [7] in that y depends only on z and not on |z| in order to match the abs-linear form used here. Then one can show the following result:

Corollary 4 (Constructive Oracle Paradigm).

For any function$f\in {\mathcal{C}}_{\mathrm{abs}}^2\left({\mathbb{R}}^n\right)$and a given point x there exist a convex polyhedral function$\check{\Delta f}(x;\Delta x)$and a concave polyhedral function$\widehat{\Delta f}(x;\Delta x)$such that

$$f(x+\Delta x)-f\left(x\right)=\frac{1}{2}\left(\check{\Delta f}(x;\Delta x)++\widehat{\Delta f}(x;\Delta x)\right)+{\mathcal{O}(\parallel \Delta x\parallel }^2)$$

Moreover, both terms and their generalized gradients at$\Delta x=0$or anywhere else can be computed with the same order of complexity as f itself.

Proof. 

In [11], we show that

$$f(x+\Delta x)-f\left(x\right)=\Delta f(x;\Delta x)+\mathcal{O}(\parallel \Delta x{\parallel }^2),$$

where $\Delta f(x;\Delta x)$ is a piecewise linearization of f developed at x and evaluated at $\Delta x.$ Applying the convex/concave decomposition of Theorem 1, one obtains immediately the assertion with a convex polyhedral function $\check{\Delta f}(x;\Delta x)$ and a concave polyhedral function $\widehat{\Delta f}(x;\Delta x)$ evaluated at $\Delta x$ The complexity results follow from the propagation rules derived so far. ☐

We had hoped that it would be possible to use this approximate decomposition into polyhedral parts to construct at least locally an exact decomposition of a general function $f\in {\mathcal{C}}_{\mathrm{abs}}^d\left({\mathbb{R}}^n\right).$ into a convex and compact part. The natural idea seems to add a sufficiently large quadratic term ${\beta (\parallel \Delta x\parallel }^2$ to

$$f(x+\Delta x)-f\left(x\right)-\frac{1}{2}\widehat{\Delta f}(x;\Delta x)=\frac{1}{2}\check{\Delta f}(x;\Delta x)+{\mathcal{O}(\parallel \Delta x\parallel }^2)$$

such that it would become convex. Then the same term could be subtracted from $\widehat{\Delta f}(x;\Delta x)$ maintaining its concavity. Unfortunately, the following simple example shows that this is not possible.

Example 1 (Half pipe).

The function

$$\begin{array}{cccc}\hfill f:{\mathbb{R}}^2& \mapsto \mathbb{R},\hfill & \hfill f(x_1,x_2)& =max(x_2^2-max(x_1,0),0)\hfill \\ \hfill & \hfill & & =\left\{\begin{array}{cc}{x}_2^2\hfill & if{x}_1\le 0\hfill \\ x_2^2-x_1\hfill & if0\le x_1\le x_2^2\hfill \\ 0\hfill & if0\le x_2^2\le x_1\hfill \end{array}\right.,\hfill \end{array}$$

(33)

in the class${\mathcal{C}}_{\mathrm{abs}}^{\infty }\left({\mathbb{R}}^n\right)$is certainly nonconvex as shown in Figure 1. As already observed in [19] this generally nonsmooth function is actually Fréchet differentiable at the origin x = 0 with a vanishing gradient $▽f\left(0\right)=0$ Hence, we have $f(\Delta x)={\mathcal{O}(\parallel \Delta x\parallel }^2$ and may simply choose constantly $\check{\Delta f}(0;\Delta x)\equiv 0\equiv \widehat{\Delta f}(0;\Delta x).$ However, neither by adding $\beta {\parallel \Delta x\parallel }^2$ nor any other smooth function to $f(\Delta x)$ can we eliminate the downward facing kink along the vertical axis $\Delta x_1=0.$ In fact, it is not clear whether this example has any DC decomposition at all.

Applying the Reverse Mode for Accumulating Generalized Gradients

Whenever gradients are propagated forward through a smooth evaluation procedure, i.e., for functions in ${\mathcal{C}}^2\left({\mathbb{R}}^n\right)$, they are uniquely defined as affine combinations of each other, starting from Cartesian basis vectors for the components of x. Given only the coefficients of the affine combinations one can propagate corresponding adjoint values, or impact factors backwards, to obtain the gradient of a single dependent with respect to all independents at a small multiple of the operations needed to evaluate the dependent variable by itself. This cheap gradient result is a fundamental principle of computational mathematics, which is widely applied under various names, for example discrete adjoints, back propagation, and reverse mode differentiation. For a historical review see [20] and for a detailed description using similar notation to the current paper see our book [5]. For good reasons, there has been little attention to the reverse mode in the context of nonsmooth analysis, where one can at best obtain subgradients. The main obstacle is again that the forward propagation rules are only sharp when all elementary operations maintain convexity, which is by the way the only constructive way of verifying convexity for a given evaluation procedure. While general affine combinations and the absolute value are themselves convex functions, they do not maintain convexity when applied to a convex argument.

The last equation of Lemma 1 shows that one cannot directly propagate a subgradient of the convex radius functions $\delta u$ because there is a reference to $v=\left|u\right|$ itself, which does not maintain convexity except when it is redundant due to its argument having a constant sign. However, it follows from the identity $\delta u=\frac{1}{2}(\check{u}-\widehat{u})$ that for all intermediates u

$$\nabla \check{u}\in \partial \check{u}\wedge \nabla \widehat{u}\in \partial \widehat{u}\Rightarrow \frac{1}{2}(\nabla \check{u}-\nabla \widehat{u})\in \partial \delta u.$$

Hence one can get affine lower bounds of the radii, although one would probably prefer upper bounds to limit the discrepancy between the convex and concave parts. When $v=\left|u\right|$ and $u=0$ we may choose according to Equation (32) any convex combination

$$\frac{1}{2}\nabla \check{v}=(1-\mu )\nabla \check{u}-\mu \nabla \widehat{u}\mathrm{for}0\le \mu \le 1.$$

(34)

It is tempting but not necessarily a good idea to always choose the weight μ equal to $\frac{1}{2}$ for simplicity.

Before discussing the reasons for this at the end of this subsection, let us note that from the values of the constants c, the intermediate values u, and the chosen weights μ it is clear how the next generalized gradient pair $(\nabla \check{v},\nabla \widehat{v})$ is computed as a linear combination of the generalized gradients of the inputs for each operation, possibly with a switch in their roles. That means after only evaluating the function f itself, not even the bounds $\check{f}$ and $\widehat{f}$, we can compute a pair of generalized gradients in $\partial \check{f}\times \partial \widehat{f}$ using the reverse mode of algorithmic differentiation, which goes back to at least [21] though not under that name. The complexity of this computation will be independent of the number of variables and relative to the complexity of the function f itself. All the operations are relatively benign, namely scaling by constants, interchanges and additions and subtractions. After all the reverse mode is just a reorganization of the linear algebra in the forward propagation of gradients. Hence, it appears that we can be comparatively optimistic regarding the numerical stability of this process.

To be specific we will indicate the (scalar) adjoint value of all intermediates $\check{u}$ and $\widehat{u}$ as usual by $\stackrel{-}{\stackrel{ˇ}{u}}\in \mathbb{R}$ and $\stackrel{-}{\widehat{u}}\in \mathbb{R}$. They are all initialized to zero except for either $\stackrel{-}{\stackrel{ˇ}{y}}=1$ or $\stackrel{-}{\widehat{y}}=1$. Then at the end of the reverse sweep, the vectors ${\left({\stackrel{-}{x}}_j\right)}_{j=1}^n$ represent either $\nabla \check{y}$ or $\nabla \widehat{y}$, respectively. For computational efficiency one may propagate both adjoint components simultaneously, so that one computes with sextuplets consisting of $\check{u},\widehat{u}$ and their adjoints with respect to $\check{y}$ and $\widehat{y}$. In any case we have the following adjoint operations. For $v=u+w$

$$(\stackrel{-}{\stackrel{ˇ}{w}},\stackrel{-}{\widehat{w}})+=(\stackrel{-}{\stackrel{ˇ}{v}},\widehat{\check{v}})\mathrm{and}(\stackrel{-}{\stackrel{ˇ}{u}},\stackrel{-}{\widehat{u}})+=(\stackrel{-}{\stackrel{ˇ}{v}},\stackrel{-}{\widehat{v}}),$$

for $v=cu$

$$(\stackrel{-}{\stackrel{ˇ}{u}},\stackrel{-}{\widehat{u}})+=\left\{\begin{array}{cc}c(\stackrel{-}{\stackrel{ˇ}{v}},\stackrel{-}{\widehat{v}})\hfill & \mathrm{if}c>0\hfill \\ (0,0)\hfill & \mathrm{if}c=0\hfill \\ c(\stackrel{-}{\widehat{v}},\stackrel{-}{\stackrel{ˇ}{v}})\hfill & \mathrm{if}c<0\hfill \end{array}\right.,$$

and finally for $v=\left|u\right|$

$$(\stackrel{-}{\stackrel{ˇ}{u}},\stackrel{-}{\widehat{u}})+=\left\{\begin{array}{cc}(2\stackrel{-}{\stackrel{ˇ}{v}}-\stackrel{-}{\widehat{v}},\stackrel{-}{\widehat{v}})\hfill & \mathrm{if}u>0\hfill \\ (-\stackrel{-}{\widehat{v}}+2(1-\mu )\stackrel{-}{\stackrel{ˇ}{v}},\stackrel{-}{\widehat{v}}-2\mu \stackrel{-}{\stackrel{ˇ}{v}})\hfill & \mathrm{if}u=0\hfill \\ (-\stackrel{-}{\widehat{v}},\stackrel{-}{\widehat{v}}-2\stackrel{-}{\stackrel{ˇ}{v}})\hfill & \mathrm{if}u<0\hfill \end{array}\right..$$

(35)

Of course, the update for the critical case $u=0$ of the absolute value is just the convex combination for the two cases $u>0$ and $u<0$ weighted by μ. Due to round-off errors it is very unlikely that the critical case $u=0$ ever occurs in floating point arithmetic. Once more, the sign of the arguments u of the absolute value function are of great importance, because they determine on which faces of the polyhedral functions $\check{f}$ and $\widehat{f}$ the current argument x is located. In some situations one prefers a gradient that is limiting in that it actually occurs as a proper gradient on one of the adjacent smooth pieces. For example, if we had simply $f\left(x\right)=v=\left|x\right|$ for $x\in \mathbb{R}$ and chose $\mu =\frac{1}{2}$ we would get $\check{v}=2\left|x\right|,\widehat{v}=0$ and find by Equation (34) that $\nabla \check{v}=2(\frac{1}{2}-\frac{1}{2})=0$ at $x=\check{x}=\widehat{x}=0$. This is not a limiting gradient of $\check{v}$ since $\partial \check{v}=[-2,2]$, whose interior contains the particular generalized gradient 0.

5. Exploiting the Convex/concave Decomposion for the DC Algorithm

In order to minimize the decomposed objective function f we may use the DCA algorithm [17] which is given in its basic form using our notation by

$$\begin{array}{c}\mathrm{Choose}{x}_0\in {\mathbb{R}}^n\\ \mathrm{For}k=0,1,2,\dots \\ \mathrm{Calculate}{g}_k\in -\partial \left(\frac{1}{2}\widehat{f}\right)\left(x_k\right)\\ \mathrm{Calculate}{x}_{k+1}\in \partial {\left(\frac{1}{2}\check{f}\right)}^{*}\left(g_k\right)\end{array}$$

where ${\left(\frac{1}{2}\check{f}\right)}^{*}$ denotes the Fenchel conjugate of $\left(\frac{1}{2}\check{f}\right)$. For a convex function $h:{\mathbb{R}}^n\mapsto \mathbb{R}$ one has

$$\begin{array}{c}\hfill w\in \partial h^{*}\left(y\right)\iff w\in \underset{x\in {\mathbb{R}}^n}{argmin}\{h\left(x\right)-y^{\top }x\},\end{array}$$

see [15], Chapter 11. Hence, the classic DCA reduces in our Euclidean scenario to a simple recurrence

$$x_{k+1}\in \underset{x\in {\mathbb{R}}^n}{argmin}\left\{\check{f}\left(x\right)+{\widehat{g}}_k^{\top }x\right\}\mathrm{for}\mathrm{some}{\widehat{g}}_k\in \partial \widehat{f}\left(x_k\right).$$

(36)

The objective function on the left hand side is a constantly shifted convex polyhedral upper bound on $2f\left(x\right)$ since

$$\check{f}\left(x\right)+{\widehat{g}}_k^{\top }x=2f\left(x\right)-\left(\widehat{f}\left(x\right)-{\widehat{g}}_k^{\top }x\right)\ge 2f\left(x\right)-\widehat{f}\left(x_k\right)+{\widehat{g}}_k^{\top }{x}_k.$$

(37)

It follows from Equation (29) and $x_{k+1}$ being a minimizer that

$$\begin{array}{ccc}\hfill f\left(x_{k+1}\right)& \le & \frac{1}{2}\left(\check{f}\left(x_{k+1}\right)+\widehat{f}\left(x_k\right)+{\widehat{g}}_k^{\top }(x_{k+1}-x_k)\right)\hfill \\ & \le & \frac{1}{2}\left(\check{f}\left(x_k\right)+\widehat{f}\left(x_k\right)\right)=f\left(x_k\right).\hfill \end{array}$$

Now, since (36) is an LOP, an exact solution $x_{k+1}$ can be found in finitely many steps, for example by a variant of the Simplex method. Moreover, we can then assume that $x_{k+1}$ is one of finitely many vertex points of the epigraph of $\check{f}$. At these vertex points, f itself attains a finite number of bounded values. Provided f itself is bounded below, we can conclude that for any choice of the ${\widehat{g}}_k\in \partial {\widehat{f}}_{{\sigma }^{\left(k\right)}}$ the resulting function values $f\left(x_k\right)$ can only be reduced finitely often so that $f\left(x_k\right)=f\left(x_{k-1}\right)$ and w.l.o.g. $x_k=x_{k-1}$ eventually. We then choose the next ${\widehat{g}}_k=\nabla {\widehat{f}}_{{\sigma }^{\left(k\right)}}$ with ${\sigma }^{\left(k\right)}={\sigma }^{(k-1)}▹\sigma \left(x_k\right)$ as the reflection of ${\sigma }^{(k-1)}$ at $\sigma \left(x_k\right)$ as defined in (15). If then again $f\left(x_{k+1}\right)=f\left(x_k\right)$ it follows from Corollary A2 that $x_k$ is a local minimizer of f and we may terminate the optimization run. Hence we obtain the DCA variant listed in Algorithm 1, which is guaranteed to reach local optimality under LIKQ. It is well defined even without this property and we conjecture that otherwise the final iterate is still a stationary point of f. The path of the algorithm on the example discussed in Section 5 is sketched in Figure 3. It reaches the stationary point $(0,-1)$ where $\sigma =(0,-1,0)$ from within the polyhedron with the signature $(-1,-1,-1)$ and then continues after the reflection $(1,-1,1)=(-1,-1,-1)▹(0,-1,0)$. From within that polyhedron the inner loop reaches the point $(1,1)$ with signature $(1,0,0)$, whose minimality is established after a search in the polyhedron ${\stackrel{-}{\mathcal{P}}}_{(1,1,-1)}$.

If the function $f\left(x\right)$ is unbounded below, so will be one of the inner convex problems and the convex minimizer should produce a ray of infinite descent instead of the next iterate $x_{k+1}$. This exceptional scenario will not be explicitly considered in the remainder of the paper. The reflection operation is designed to facilitate further descent or establish local optimality. It is discussed in the context of general optimality conditions in the following subsection.

Algorithm1 Reflection DCA

Require:$x_0\in {\mathbb{R}}^n,$ 1: Set $f_{-1}=\infty$ and Evaluate $f_0=f\left(x_0\right)$ 2: for$k=0,1,\dots$do 3:     if $f_k<f_{k-1}$ then                    ▹ Normal iteration with function reduction 4:          Choose $0\notin \sigma \succ \sigma \left(x_k\right)$                 ▹ Here different heuristics may be applied 5:          Compute ${\widehat{g}}_k=\nabla {\widehat{f}}_{\sigma }$                      ▹ Apply formula of Corollary 1 6:     else                           ▹ The starting point was already optimal 7:          Reflect $\tilde{\sigma }=\sigma ▹\sigma \left(x_k\right)$                 ▹ The symbol ▹ is defined in Equation (15). 8:          Update ${\widehat{g}}_k=\nabla {\widehat{f}}_{\tilde{\sigma }}$ 9:     end if 10:      Calculate $x_{k+1}\in argmin\left\{\left.\check{f}\left(x\right)+{\widehat{g}}_k^{\top }x\right|x\in {\mathbb{R}}^n\right\}$         ▹ Apply any LOP finite solver 11:   Set $f_{k+1}=f\left(x_{k+1}\right)$ 12:   if$f_{k+1}=f_k=f_{k-1}$ then                      ▹ Local optimality established 13:          Stop 14:     end if 15: endfor

5.1. Checking Optimality Conditions

Stationarity of $x_k$ happens when the convex function $\check{f}\left(x\right)+{\widehat{g}}_k^{\top }x$ is minimal at $x_k$ so that for all large k

$$0\in \partial \check{f}\left(x_k\right)+{\widehat{g}}_k\iff {\widehat{g}}_k\in \partial \widehat{f}\left(x_k\right)\cap (-\partial \check{f}\left(x_k\right))\ne \varnothing .$$

(38)

The nonemptiness condition on the right hand side is known as criticality of the DC decomposition at $x_k$, which is necessary but not sufficient even for local optimality of $f\left(x\right)$ at $x_k$. To ensure the latter one has to verify that all ${\widehat{g}}_k\in \partial \widehat{f}\left(x_k\right)$ satisfy the criticality condition (38) so that

$$\partial \widehat{f}\left(x_k\right)\subset -\partial \check{f}\left(x_k\right)\iff {\partial }^L\widehat{f}\left(x_k\right)\subset -\partial \check{f}\left(x_k\right).$$

(39)

The left inclusion is a well known local minimality condition [22], which is already sufficient in the piecewise linear case. The right inclusion is equivalent to the left one due to the convexity of $\partial \check{f}\left(x_k\right)$.

If $\check{f}$ and $\widehat{f}$ were unrelated convex and concave polyhedral functions, one would normally consider it extremely unlikely that $\widehat{f}$ were nonsmooth at any one of the finitely many vertices of the polyhedral domain decomposition of $\check{f}$. For instance when $\widehat{f}$ is smooth at $x_k$ we find that $\partial \widehat{f}\left(x_k\right)=\left\{{\widehat{g}}_k\right\}$ is a singleton so that criticality according to Equation (38) is already sufficient for local minimality according to Equation (39). As we have seen in Theorem 1 the two parts have exactly the same switching structure. That means they are nonsmooth on the same skeleton of lower dimensional polyhedra. Hence, neither ${\partial }^L\check{f}\left(x_k\right)$ nor ${\partial }^L\widehat{f}\left(x_k\right)$ will be singletons at minimizing vertices of the upper bound so that checking the validity of Equation (39) appears to be a combinatorial task at first sight.

However, provided the Linear Independence Kink Qualification (LIKQ) defined in [7] is satisfied at the candidate minimizer $x_k$, the minimality can be tested with cubic complexity even in case of a dense abs-linear form. Moreover, if the test fails one can easily calculate a descent direction d. The details of the optimality test in our context including the calculation of a descent direction are given in the Appendix A. They differ slightly from the ones in [7]. Rather than applying the optimality test Proposition A1 explicitly, one can use its Corollary A2 stating that if $\mathring{x}$ with $\mathring{\sigma }=\sigma \left(\mathring{x}\right)$ is a local minimizer of the restriction of f to a polyhedron ${\stackrel{-}{\mathcal{P}}}_{\sigma }$ with definite $\sigma \succ \mathring{\sigma }$ then it is a local minimizer of the unrestricted f if and only if it also minimizes the restriction of f to ${\stackrel{-}{\mathcal{P}}}_{\tilde{\sigma }}$ with the reflection $\tilde{\sigma }=\sigma ▹\mathring{\sigma }$. The latter condition must be true if $\mathring{x}$ also minimizes $f\left(x\right)+\nabla {\widehat{f}}_{\tilde{\sigma }}$, which can be checked by solving that convex problem. If that test fails the optimization can continue.

5.2. Proximal Rather Than Global

By some authors the DCA algorithm has been credited with being able to reach global minimizers with a higher probability than other algorithms. There is really no justification for this optimism in the light of the following observation. Suppose the objective $f\left(x\right)=\frac{1}{2}(\check{f}\left(x\right)+\widehat{f}\left(x\right))$ has an isolated local minimizer $x_{*}$. Then there exists an $\epsilon >0$ such that the level set $\{x\in {\mathbb{R}}^n:f\left(x\right)\le f\left(x_{*}\right)+\epsilon \}$ has a bounded connected component containing $x_{*}$, say ${\mathcal{L}}_{\epsilon }$. Now suppose DCA is started from any point $x_0\in {\mathcal{L}}_{\epsilon }$. Since $f_0\left(x\right)\equiv \frac{1}{2}(\check{f}\left(x\right)+\widehat{f}\left(x_0\right)+\widehat{g}{\left(x_0\right)}^{\top }(x-x_0))$ is by Equation (37) a convex upper bound on $f\left(x\right)$ its level set $\{f_0\left(x\right)\le f\left(x_0\right)\}$ will be contained in ${\mathcal{L}}_{\epsilon }$. Hence any step from $x_0$ that reduces the upper bound $f_0\left(x\right)$ must stay in the same component, so there is absolutely no chance to move away from the catchment ${\mathcal{L}}_{\epsilon }$ of $x_0$ towards another local minimizer of f, whether global or not. In fact, by adding the convex term

$$\frac{1}{2}\left(\widehat{f}\left(x_0\right)+\widehat{g}{\left(x_0\right)}^{\top }(x-x_0)-\widehat{f}\left(x\right)\right)\ge 0,$$

which vanishes at $x_0$, to the actual objective $f\left(x\right)$ one performs a kind of regularization, like in the proximal point method. This means the step is actually held back compared to a larger step that might be taken by a method that only requires the reduction of $f\left(x\right)$ itself.

Hence we may interpret DCA as a proximal point method where the proximal term is defined as an affinely shifted negative of the concave part. Since in general the norm and the coefficient defining the proximal term may be quite hard to select, this way of defining it may make a lot of sense. However, it is certainly not global optimization. Notice that in this argument we have used neither the polyhedrality nor the inclusion property. So it applies to a general DC decomposition on Euclidean space. Another conclusion from the "holding back" observation is that it is probably not worthwhile to minimize the upper bound very carefully. One might rather readjust the shift ${\widehat{g}}^{\top }x$ after a few or even just one iteration.

6. Nesterov’s Piecewise Linear Example

According to [6], Nesterov suggested three Rosenbrock-like test functions for nonsmooth optimization. One of them given by

$$\begin{array}{cc}\hfill f\left(x\right)=& \frac{1}{4}|x_1-1|+\sum _{i=1}^{n-1}\left|x_{i+1}-2\left|x_i\right|+1\right|\hfill \end{array}$$

(40)

is nonconvex and piecewise linear. It is shown in [6] that this function has $2^{n-1}$ Clarke stationary points only one of which is a local and thus the global minimizer. Numerical studies showed that optimization algorithms tend to be trapped at one of the stationary points making it an interesting test problem. We have demonstrated in [23] that using an active signature strategy one can guarantee convergence to the unique minimizer from any starting point albeit using in the worst case $2^n$ iterations as all stationary points are visited. Let us first write the problem in the new abs-linear form.

Defining the $s=2n$ switching variables

$$z_i=F_i(x,|z\left|\right)=x_i\mathrm{for}1\le i<n,z_n=F_n(x,|z\left|\right)=x_1-1,$$

and

$$z_{n+i}=F_{n+i}(x,|z\left|\right)=x_{i+1}-2|z_i|+1\mathrm{for}1\le i<n,z_s=\frac{1}{4}|z_n|+\sum _{i=1}^{n-1}\left|z_{n+i}\right|$$

the resulting objective function is then simply identical to $y=f\left(x\right)=z_s$. With the vectors and matrices

$$\begin{array}{ccc}\hfill c^{\top }& =& (0,-1,e_{n-1}^{\top },0)\in {\mathbb{R}}^{(n-1)+1+(n-1)+1},Z=\left[\begin{array}{cc}{I}_{n-1}& 0\\ I_{n-1}& 0\\ 0& 1\\ 0& 0\end{array}\right]\in {\mathbb{R}}^{s\times (n-1)+1},\hfill \\ \hfill M& =& 0,L=\left[\begin{array}{cccc}0& 0& 0& 0\\ 0& 0& 0& 0\\ -2I_{n-1}& 0& 0& 0\\ 0& \frac{1}{4}& e_{n-1}^{\top }& 0\end{array}\right]\in {\mathbb{R}}^{s\times (n-1)+1+(n-1)+1},d=0\in \mathbb{R},\hfill \\ \hfill a& =& 0,b^{\top }=(0,\cdots ,0,1)\in {\mathbb{R}}^{(2n-1)+1},\hfill \end{array}$$

where Z and L have different row partitions, one obtains an abs-linear form (11) of f. Here, $I_k$ denotes the identity matrix of dimension k, $e^{\top }=(1,\cdots ,1)\in {\mathbb{R}}^k$ the vector containing only ones and the symbol 0 pads with zeros to achieve the specified dimensions. One can easily check that ${\left|L\right|}^2\ne 0={\left|L\right|}^3$, hence this example has switching depth $\nu =2$. The geometry of the situation is depicted in Figure 3, which was already briefly discussed in Section 3 and Section 5.

Since the corresponding extended abs-linear form for $\tilde{f}=(y,\delta y)$ does not provide any new insight we do not state it here. Directly in terms of the original equations we obtain for the radii

$$\delta z_i=0\mathrm{for}1\le i\le n,\delta z_{n+i}=2|z_i|=2|x_i|\mathrm{for}1\le i<n$$

(41)

and

$$\begin{array}{ccc}\hfill \delta f=\delta z_s& =& \frac{1}{4}|z_n|+\sum _{i=1}^{n-1}\left(\right|z_{n+i}\left|+2\delta z_{n+i}\right)\hfill \\ & =& \frac{1}{4}|x_1-1|+\sum _{i=1}^{n-1}\left(\right|x_{i+1}-2|x_i|+1|+4|x_i\left|\right).\hfill \end{array}$$

(42)

Thus, from Equation (7) we get the convex and concave part explicitly as

$$\begin{array}{cc}\hfill {\check{z}}_i& =z_i={\widehat{z}}_i\mathrm{for}1\le i\le n,\hfill \\ & \left.\begin{array}{c}{\check{z}}_{n+i}=x_{i+1}+1\hfill \\ {\widehat{z}}_{n+i}=x_{i+1}-4|z_i|+1=x_{i+1}-4\left|x_i\right|+1\hfill \end{array}\right\}\mathrm{for}1\le i<n\hfill \end{array}$$

and most importantly

$$\begin{array}{cc}\hfill \check{f}& =z_s+\delta z_s=\frac{1}{2}|x_1-1|+2\sum _{i=1}^{n-1}\left(\left|x_{i+1}^{}-2\left|x_i\right|+1\right|+2\left|x_i\right|\right)\hfill \\ \hfill \widehat{f}& =z_s-\delta z_s=-4\sum _{i=1}^{n-1}\left|x_i\right|.\hfill \end{array}$$

Clearly $\widehat{f}$ is a concave function and to check the convexity of $\check{f}$ we note that

$$\begin{array}{ccc}\hfill \left|x_{i+1}^{}-2\left|x_i\right|+1\right|+2\left|x_i\right|& =& \left|2|x_i{|}^{}-1-x_{i+1}^{}\right|+\left(2|x_i|-1-x_{i+1}^{}\right)+x_{i+1}+1\hfill \\ & =& 1+x_{i+1}^{}+2max\left(0,2|x_i|-x_{i+1}^{}-1\right).\hfill \end{array}$$

(43)

The last expression is the sum of an affine function and the positive part of the sum of the absolute value and an affine function, which must therefore also be convex. The corresponding term in Equation (42) is the same with the convex function $2|x_i|$ added, so that $\delta f$ is also convex in agreement with the general theory. Finally, one verifies easily that

$$\begin{array}{c}\hfill \widehat{f}\le f=\frac{1}{2}(\check{f}+\widehat{f})\le \check{f},\end{array}$$

which is the whole idea of the decomposition. It would seem that the automatic decomposition by propagation through the abs-linear procedure yields a rather tight result. The function f as well as the lower and upper bound given by the convex/concave decomposition are illustrated on the left hand side of Figure 2. Notice that the switching structure is indeed identical for all three as stated in Theorem 1. On the right hand side of Figure 2, the difference $2\delta f$ between the upper, convex and lower, concave bound is shown, which is indeed convex.

It is worthwhile to look at the condition number of the decomposition, namely we get the following trivial bound

$$\begin{array}{ccc}\hfill \kappa (\check{f},\widehat{f})& =& \underset{x\in {\mathbb{R}}^n}{sup}\frac{\frac{1}{2}|x_1-1\left|+2{\sum }_{i=1}^{n-1}\left(\left|x_{i+1}^{}-2\left|x_i\right|+1\right|+4|x_i\right|\right)}{\frac{1}{2}|x_1-1|+2{\sum }_{i=1}^{n-1}|x_{i+1}-2|x_i|+1|}\hfill \\ & =& 1+\underset{x\in {\mathbb{R}}^n}{sup}\frac{8{\sum }_{i=1}^{n-1}\left|x_i\right|}{\frac{1}{4}|x_1-1|+2{\sum }_{i=1}^{n-1}|x_{i+1}-2|x_i|+1|}=\infty .\hfill \end{array}$$

The disappointing right hand side value follows from the fact that at the well known unique global optimizer $x_{*}=(1,1,\dots ,1)\in {\mathbb{R}}^n$ the numerator is zero and the denominator positive. However, elsewhere, we can bound the conditioning as follows.

Lemma 3.

In case of the example (40) there is a constant $c\in \mathbb{R}$ such that

$$\kappa (\check{f}\left(x\right),\widehat{f}\left(x\right))\le 1+\frac{c}{min(\parallel x-x_{*}\parallel ,3)}.$$

(44)

Proof. 

Since the denominator is piecewise linear and vanishes only at the minimizer $x_{*}$ there must be a constant $c_0>0$ such that for $\parallel x-x_{*}{\parallel }_{\infty }\le 3$

$$\frac{8{\sum }_{i=1}^{n-1}\left|x_i\right|}{\frac{1}{4}|x_1-1|+2{\sum }_{i=1}^{n-1}|x_{i+1}-2|x_i|+1|}\le \frac{8{\sum }_{i=1}^{n-1}\left|x_i\right|}{c_0{\parallel x-x_{*}\parallel }_{\infty }}\le \frac{{8(n-1)\parallel x\parallel }_{\infty }}{c_0{\parallel x-x_{*}\parallel }_{\infty }}\le \frac{32(n-1)}{c_0{\parallel x-x_{*}\parallel }_{\infty }},$$

which takes the value $32(n-1)/\left(3c_0\right)$ on the boundary. On the other hand we get for ${\parallel x\parallel }_{\infty }\ge 2$ and thus in particular $\parallel x-x_{*}{\parallel }_{\infty }\ge 3$

$$\frac{8{\sum }_{i=1}^{n-1}\left|x_i\right|}{\frac{1}{4}|x_1-1|+2{\sum }_{i=1}^{n-1}|x_{i+1}-2|x_i\left|+1\right|}\le \frac{{4(n-1)\parallel x\parallel }_{\infty }}{{max}_{1\le i<n}\left|2\right|x_i|-x_{i+1}-1|}\le \frac{4(n-1)}{2-1-1/2}\le 8(n-1).$$

Assuming without loss of generality that $c_0\le 4/3$ we can combine the two bounds to obtain the assertion with $c\equiv 32(n-1)/c_0$. ☐

Hence, we see the condition number $\kappa (\check{f}\left(x\right),\widehat{f}\left(x\right))$ is nicely bounded and the decomposition should work as long as our optimization algorithm has not yet reached its goal $x_{*}$. It is verified in the companion article [24], that the DCA exploiting the observations made in this paper reaches the global minimizer in finitely many steps. It was already shown in [7] that the LIKQ condition is satisfied everywhere and that the optimality test singles out the unique minimizer correctly. In Figure 3, the arrows indicate the path of our reflection version of the DCA method as described in Section 5.

7. Summary, Conclusions and Outlook

In this paper the following new results were achieved

• For every piecewise linear function f given as an abs-linear evaluation procedure, rules for simultaneously evaluating its representation as the average of a concave lower bound $\widehat{f}$ and a convex upper bound $\check{f}$ are derived.
• The two bounds can be constructively expressed as a single maximum and minimum of affine functions, which drastically simplifies the classical $min-max$ representation. Due to its likely combinatorial complexity we do not recommend this form for practical calculations.
• For the two bounds $\check{f}$ and $\widehat{f}$, generalized gradients $\check{g}$ and $\widehat{g}$ can be propagated forward or reverse through the convex or concave operations that define them. The gradients are not unique but guaranteed to yield supporting hyperplanes and thus provide a verified version of the oracle paradigm.
• The DCA algorithm can be implemented such that a local minimizer is reached in finitely many iterations, provided the Linear Independence Kink Qualification (LIKQ) is satisfied. It is conjectured that without this assumption the algorithm still converges in finitely many steps to a Clarke stationary point. Details on this can be found in the companion paper [24].

These results are illustrated on the piecewise linear Rosenbrock variant of Nesterov.

On a theoretical level it would be gratifying and possibly provide additional insight, to prove the result of Corollary A3 directly using the explicit representations of the generalized differentials of the convex and concave part given in Corollary 1. Moreover, it remains to be explored what happens when LIKQ is not satisfied. We have conjectured in [25] that just verifying the weaker Mangasarian Fromovitz Kink Qualification (MFKQ) represents an NP hard task. Possibly, there are other weaker conditions that can be cheaply verified and facilitate the testing for at least local optimality.

Global optimality can be characterized theoretically in terms of $\epsilon -$subgradients, albeit with ε arbitrarily large [26]. There is the possibility that the alternative definition of ε-gradients given in [18] might allow one to constructively check for global optimality. It does not really seem clear how these global optimality conditions can be used to derive corresponding algorithms.

The implementation of the DCA algorithm can be optimized in various ways. Notice that for applying the Simplex method in standard form, one could use for the representation as DC function the max-part in the more economical representation Equation (27) introducing $\overline{m}$ additional variables, rather than the potentially combinatorial Equation (28) to assemble the constraint matrix. In any case it seems doubtful that solving each sub problem to completion is a good idea, especially as the resulting step in the outer iteration is probably much too small anyhow. Therefore, the generalized gradient of the concave part, which defines the inner problem, should probably be updated much more frequently. Moreover, the inner solver might be an SQOP type active signature method or a matrix free gradient method with momentum term, as is used in machine learning, notwithstanding the nonsmoothness of the objective. Various options in that range will be discussed and tested in the companion article [24].

Finally, one should always keep in mind that the task of minimizing a piecewise linear function will most likely occur as an inner problem in the optimization of a piecewise smooth and nonlinear function. As we have shown in [27] the local piecewise linear model problem can be obtained easily by a slight generalization of automatic or algorithmic differentiation, e.g., ADOL-C [28] and Tapenade [29].