A Customized ADMM Approach for Large-Scale Nonconvex Semidefinite Programming

Abstract

We investigate a class of challenging general semidefinite programming problems with extra nonconvex constraints such as matrix rank constraints. This problem has extensive applications, including combinatorial graph problems, such as MAX-CUT and community detection, reformulated as quadratic objectives over nonconvex constraints. A customized approach based on the alternating direction method of multipliers (ADMM) is proposed to solve the general large-scale nonconvex semidefinite programming efficiently. We propose two reformulations: one using vector variables and constraints, and the other further reformulating the Burer–Monteiro form. Both formulations admit simple subproblems and can lead to significant improvement in scalability. Despite the nonconvex constraint, we prove that the ADMM iterates converge to a stationary point in both formulations, under mild assumptions. Additionally, recent work suggests that in this matrix form, when the matrix factors are wide enough, the local optimum with high probability is also the global optimum. To demonstrate the scalability of our algorithm, we include results for MAX-CUT, community detection, and image segmentation.

1. Introduction

We consider rank-constrained semidefinite optimization problems (SDPs) of the type:

$$\begin{array}{ccc}\hfill \underset{Z,X}{\min }& & f\left(Z\right),\hfill \\ \hfill s.t.& & \mathcal{A}\left(Z\right)=b,Z=X{X}^T,X\in \mathcal{C},\hfill \end{array}$$

(1)

where the matrix variable $Z\in {\mathbb{S}}_{+}^n$ is an $n\times n$ symmetric semidefinite matrix, and $X\in {\mathbb{R}}^{n\times r}$ a low-rank symmetric factor. The linear constraints $\mathcal{A}\left(Z\right)=b$ constrain either the diagonal or trace of Z, and the set $\mathcal{C}$ controls desirable features of the factor—e.g., nonnegativity, integer, norm-1, etc. ($\mathcal{C}$ may be nonconvex.) The objective function $f\left(x\right)$ is convex, differentiable everywhere, with $L_f$-Lipschitz gradient, but the overall problem (1) is nonconvex.

This problem is equivalent to many important nonconvex SDPs, such as the MAX-CUT problem and its related applications [1,2,3], rank-constrained nonnegative matrix factorization problem [4,5], and constrained eigenvalue problems [6,7,8]. It is known that exactly solving (1) globally is in general a very difficult problem, as it includes many NP-hard problems. Methods for heuristically solving (1) fall in three categories: (i) solving the convexified SDP, where (1) does not have the rank-r or $X\in \mathcal{C}$ constraint, using any convex optimization method [9,10,11], (ii) approximately solving (1) using an alternating minimization method [12,13] and relying on statistical arguments suggesting that the acquired local optimal = the global optimal [13], or (iii) using other application-specific approaches [2,14]. The methods investigated in this paper fall in the second category. Specifically, we investigate solving (1) using ADMM and linearized ADMM on two reformulations. We find that these flexible reformulations allow easy incorporation of low-rank and sparse structures, making the resulting algorithm extremely scalable in both memory and computation, which we demonstrate on a number of popular applications.

However, often nonconvex formulations of SDPs are not favored because the convergence behavior of standard algorithms is not well understood. Specifically, an iterative procedure can do one of four things: diverge, oscillate within a bounded interval, converge to an arbitrary point, or converge to a useful point. We show that linearized ADMM on a nonsymmetric reformulation of (1) can either converge to a stationary point or diverge to $\pm \infty$; it cannot oscillate or converge to a non-stationary point. Additionally, for the case without linear constraints, vanilla ADMM is guaranteed to converge to a stationary point with a monotonically decreasing augmented Lagrangian term, and at a linear rate if the objective is strongly convex.

2. Applications

It is well known that many convex optimization problems can be reformulated as SDPs (e.g., [15]). In nonconvex optimization, SDPs are studied in several key areas, as tight convex relaxations of otherwise NP-hard problems.

2.1. Combinatorial Problems

A simple reparameterization of the constraint $x\in {\mathbb{R}}^n$, $x_i\in \{-1,1\}$ is as $X=x{x}^T$, $\mathbf{diag}\left(X\right)=1$. This property has been heavily exploited for finding lower bounds in combinatorial optimization [9,16,17] and generalized further to polynomial optimization [18,19]. Of high interest is the MAX-CUT problem:

$$\underset{x\in {\mathbb{R}}^n}{\min }{x}^{T}Cx,s.t.x_i\in \{-1,1\},i=1,\cdots ,n,$$

(2)

where $C=(A-\mathbf{diag}(A\mathbf{1}\left)\right)/4$ and $A\in {\mathbb{S}}^n$ is the symmetric adjacency matrix of an undirected graph. Written in this way, the solution to (1) is exactly the maximum cut of an undirected graph with nonnegative weights $A_{ij}$.

This seemingly simple framework appears in many other applications, such as community detection [20] and image segmentation [21], and is equivalent to the nonconvex SDP:

$$\underset{Z}{\min }\mathbf{Tr}\left(CZ\right),s.t.Z_{kk}=1,Z⪰0,\mathbf{rank}\left(Z\right)=1.$$

(3)

Lifting $x\in {\mathbb{R}}^n$ to a skinny matrix $X\in {\mathbb{R}}^{n\times k}$ generalizes this technique to partitioning [22] and graph coloring problems [23].

2.1.1. Related Works on MAX-CUT

More generally, combinatorial methods can be solved using branch-and-bound schemes, using a linear relaxation of (1) as a bound [24,25], where the binary constraint $x\in \{-1,1\}$ is relaxed to $0\le (x+1)/2\le 1$. Historically, these “polyhedral methods” were the main approach to finding exact solutions to the MAX-CUT problem. Though this is an NP(non-deterministic polynomial-time)-hard problem, if the graph is sparse enough, branch-and-bound converges quickly even for very large graphs [25]. However, when the graph is not very sparse, the linear relaxation is loose, and finding efficient branching mechanisms is challenging, causing the algorithm to run slowly. The MAX-CUT problem can also be approximated by one pass of the linear relaxation (with bound $\frac{f_{\mathrm{relax}}}{f_{\mathrm{exact}}}\ge 2\times \#$edges) [26].

A tighter approximation can be found with the semidefinite relaxation, which is also used for better bounding in branch-and-bound techniques [27,28,29,30,31]. In particular, the rounding algorithm of [9] returns a feasible $\widehat{x}$ given optimal Z, and is shown in expectation to satisfy $\frac{x^{T}Cx}{{\widehat{x}}^{T}C\widehat{x}}\ge 0.878$. For this reason, the semidefinite relaxation for problems of type (1) is heavily studied (e.g., [11,32,33]).

2.1.2. Specialization to Community Detection

A small modification of the matrix C generalizes problems of form (2) and (3) to community detection in machine learning. Here, the problem is to identify node clusters in undirected graphs that are more likely to be connected with each other than with nodes outside the cluster. This prediction is useful in many graphical settings, such as interpreting online communities through social networks or linking behavior [34], interpreting biological ecosystems [35], finding disease sources in epidemiology [36], and many more. There are many varieties and methodologies in this field, and it would be impossible to list them all, though many comprehensive overviews exist (e.g., [2]).

The stochastic binary model [37] is one of the simplest generative models for this application. Given a graph with n nodes and parameters $0<q<p<1$, the model partitions the nodes into two communities and generates an edge between nodes in a community with probability p and nodes in two different communities with probability q. Following the analysis in [20], we can define $C=\frac{p+q}{2}\mathbf{1}{\mathbf{1}}^T-A$, where A is the graph adjacency matrix, and the solution to (1) gives a solution to the community detection problem with sharp recovery guarantees.

2.2. Nonnegative Factorization

For a symmetric matrix C, the maximum eigenvalue/eigenvector pair of C is the solution to the nonconvex optimization problem:

$$\underset{x\in {\mathbb{R}}^n}{\max }{x}^{T}Cx,s.t.{\parallel x\parallel }_2=1.$$

(4)

By inverting the sign of C, we can transform this into a minimization problem or equivalently acquire the minimum eigenvalue/eigenvector pair. Interestingly, despite the nonconvex nature of (4), we have many efficient globally optimal methods for finding x, e.g., Lanczos, Arnoldi, etc. However, adding any additional constraints, such as nonnegativity of x [38], and simple methods generally do not work without heavy data assumptions [39]. This is of interest in problems such as phase retrieval, recommender systems with positive-only observations, clustering and topic models, etc. Here, we discuss three variations of the nonnegative factorization problem appearing in the literature, all of which are special instances of (1).

2.2.1. Optimization over Spectrahedron

We can frame (4) as a linear objective over the spectrahedron:

$$\underset{Z\in {\mathbb{S}}^n}{\min }\mathbf{Tr}\left(CZ\right),s.t.\mathbf{Tr}\left(Z\right)=1,Z⪰0.$$

(5)

If additionally the maximum eigenvalue of C is isolated (corresponding only to one leading eigenvector), then $Z=x{x}^T$ and $Cx={\lambda }_{max}\left(C\right)x$. To see this, by definition,

$$\begin{array}{ccc}\hfill {\lambda }_{max}\left(C\right)& & =\underset{{x:\parallel x\parallel }_2=1}{max}{x}^{T}Cx\hfill \\ & & =\underset{Z:Z=x{x}^T,{\parallel x\parallel }_2=1}{max}\mathbf{Tr}\left(CZ\right)\hfill \\ & & =\underset{Z:\mathbf{Tr}\left(Z\right)=1,X⪰0}{max}\mathbf{Tr}\left(CZ\right).\hfill \end{array}$$

(6)

As a consequence, note that though (5) is convex, the solution $Z^{*}$ will always have rank 1 when ${\lambda }_{max}\left(C\right)$ has multiplicity 1. A simple extension of (5) often used in nonnegative PCA [40] is:

$$\begin{array}{c}\underset{Z\in {\mathbb{S}}^n,x\in {\mathbb{R}}^n}{\min }\mathbf{Tr}\left(CZ\right),\hfill \\ s.t.\mathbf{Tr}\left(Z\right)=1,Z⪰0,Z=x{x}^T,x\ge 0,\hfill \end{array}$$

(7)

which is an instance of (1) with $\mathcal{C}$ the nonnegative orthant.

2.2.2. Factorization with Partial Observations

An equivalent way of formulating the top-k nonnegative-eigenvector problem is as the nonnegative minimizer X to $\parallel X{X}^T{-C\parallel }_2$ where X is ${\mathbb{R}}^{n\times k}$. However, in many applications, we may not have full view of the matrix C, (e.g., C is a rating matrix). Suppose that an index set $\mathsf{\Omega }$ defines the observed entries, e.g., $\{i,j\}\in \mathsf{\Omega }$ implies that $C_{ij}$ is known. Then, the nonnegative factorization problem can be written as:

$$\begin{array}{c}\hfill {\displaystyle \underset{Z\in {\mathbb{S}}^n,x\in {\mathbb{R}}^n}{\min }\sum _{i,j\in \mathsf{\Omega }}{(Z_{ij}-C_{ij})}^2,}\\ \hfill s.t.Z=x{x}^T,x\ge 0.\end{array}$$

(8)

This formulation exists in [41].

2.2.3. Projective Nonnegative Matrix Factorization

A third method toward this goal is to optimize over the low-rank projection matrix itself [42], a variant of nonnegative matrix factorization, solving:

$$\begin{array}{c}\hfill \underset{Z\in {\mathbb{S}}^n,X\in {\mathbb{R}}^{n\times k}}{min}{\parallel B-ZB\parallel }_2,\\ \hfill s.t.Z=X{X}^T,X\ge 0,\end{array}$$

(9)

Here, the data matrix may not even be symmetric, but $\frac{1}{\mathbf{Tr}\left(Z\right)}ZB$ will approximate the projection of B to its top-k singular vectors.

3. Related Work

3.1. Convex Relaxations

If $r=n$ and $\mathcal{C}={\mathbb{S}}^n$, then (1) is a convex problem, and can be solved using many conventional methods with strong convergence guarantees. However, even in this case, if n is large, traditional semidefinite solvers are computationally limiting. In the most general case, an interior point method solves at each iteration a KKT system of at least order $n^6$, and most first-order methods for general SDPs require eigenvalue decompositions, which are of order $O\left(n^3\right)$ per iteration.

3.2. Low-Rank Convex Cases

In fact, assuming low-rank solutions often allows for the construction of faster SDP methods. In [43], it is noted that the rank of the primal PSD matrix variable is equal to the multiplicity of the matrix variable arising from the gauge dual formulation, and finding only those r corresponding eigenvectors can recover the primal solution. In [10], a similar observation is made of the Lagrange dual variable and thus the dual problem can be solved via a modified bundle method. More generally, the recently popularized conditional gradient algorithm (also called the Frank–Wolfe algorithm) efficiently solves norm-constrained problems for nonsymmetric matrices [44], exploiting the fact that the dual norm minimizer can be computed efficiently; see also [45,46,47].

3.3. Nonconvex Cases

In close connection with these observations, [12,48] proposed simply reformulating semidefinite matrix variables $Z=X{X}^T$, solving the “standard” nonconvex SDP:

$$\underset{X\in {\mathbb{R}}^{n\times r}}{\min }\langle C,X{X}^T\rangle ,s.t.\mathcal{A}\left(X{X}^T\right)=b,$$

(10)

by sequentially optimizing the Lagrangian. However, solving (1) is still numerically burdensome; in the augmented Lagrangian term, the objective is quartic in R, and is usually solved using an iterative numerical method, such as L-BFGS.

3.4. Global Optimality of a Nonconvex Problem with Linear Objective

The main motivation behind solving rank-constrained problems using convex optimization methods comes from key results in  [49,50] which show that for a linear SDP, when $X^{*}$ is the optimum and $r=\mathbf{rank}\left(X^{*}\right)$, then $\frac{r(r+1)}{2}\ge m$ where m is the number of linear constraints. Furthermore, a recent work [13] shows that almost all local optima of FSDP are also global optima, suggesting that any stationary point of the FSDP is also a reasonable approximation of (1), if the constraint space of (10) is compact and sufficiently smooth, e.g., $A_{i}Y$ linearly independent whenever $\langle A_i,Y{Y}^T\rangle =b_i$ for all $i=1,\cdots ,m$. The MAX-CUT problem satisfies this constraint; an example of a linear SDP without this condition is the phase retrieval problem [51], when $m>n$.

3.5. Nonconvex Constraint $\mathcal{C}$

Although there are many cases where the linear constraint in (1) serves a distinct purpose, largely it is introduced to tighten the convex relaxation. When working in the nonconvex formulation, for many applications, the linear constraint becomes superfluous, and a more useful reformulation may be:

$$\underset{x,y}{\min }g\left(x\right),s.t.x=y,y\in \mathcal{C},$$

for some nonconvex set $\mathcal{C}$ (e.g., $\mathcal{C}={\{-1,1\}}^n$). Note that the projection on $\mathcal{C}$ is extremely easy, despite its nonconvexity. Although less explored, this idea is not new; see [52] (chapter 9).

3.6. ADMM for Nonconvex Problems

The alternating direction method of multipliers (ADMM) [53,54] is now a popular method [52] for convex large-scale distributed optimization problems with understood convergence rates [55] and variations [56,57,58]. It is closely related to dual decomposition methods, but alternates its subproblems, and makes use of augmented Lagrangians, which smooths the subproblems and reduces the influence of the dual ascent step size. Although there are extensions to many variable blocks, most ADMM implementations use two variable block decompositions, solving:

$$\underset{x}{min}g\left(x\right)+h\left(y\right),s.t.Ax=By,$$

by alternatingly minimizing over each variable in the augmented Lagrangian:

$${\mathcal{L}}_{\rho }(x,y;u)=g\left(x\right)+h\left(y\right)+u^T(Ax-By)+\frac{\rho }{2}{\parallel Ax-By\parallel }_2^2,$$

and then incrementally updating the dual variable:

$$\begin{array}{ccc}\hfill x^{+}& =& arg\underset{x}{min}{\mathcal{L}}_{\rho }(x,y;u),\hfill \\ \hfill y^{+}& =& arg\underset{x}{min}{\mathcal{L}}_{\rho }(x^{+},y;u),\hfill \\ \hfill u^{+}& =& u+\rho (A{x}^{+}-B{y}^{+}).\hfill \end{array}$$

Here, any $\rho >0$ will achieve convergence.

In general, there is a lack of theoretical justification for ADMM on nonconvex problems despite its good numerical performance. Almost all works concerning ADMM on nonconvex problems investigate when nonconvexity is in the objective functions ([59,60,61,62,63], and also [64,65] for matrix factorization). Under a variety of assumptions (e.g., convergence or boundedness of dual objectives) they are shown to converge to a KKT stationary point.

In comparison, relatively fewer works deal with nonconvex constraints. Ref. [66] tackles polynomial optimization problems by minimizing a general objective over a spherical constraint ${\parallel x\parallel }_2=1$, Ref. [67] solves general QCQPs, and Ref. [68] solves the low-rank-plus-sparse matrix separation problem. In all cases, they show that all limit points are also KKT stationary points, but do not show that their algorithms will actually converge to the limit points. In this work, we investigate a class of nonconvex constrained problems and show with much milder assumptions that the sequence always converges to a KKT stationary point.

We now present our main results, the algorithms, and convergence analysis for different formulations.

4. Linearized ADMM on Full SDP

We first investigate a reformulation of (1) as:

$$\begin{array}{ccc}\hfill \underset{Z,X,Y}{\min }& & f\left(Z\right)+{\delta }_{\left\{0\right\}}(\mathcal{A}\left(Z\right)-b)+{\delta }_{\mathcal{C}}\left(Y\right),\hfill \\ \hfill s.t.& & Z={\left(X{Y}^T\right)}_{\mathsf{\Omega }},X=Y,\hfill \end{array}$$

(11)

with variables $Z\in {\mathbb{S}}^{n\times n}$, $X\in {\mathbb{R}}^{n\times r}$, and $Y\in {\mathbb{R}}^{n\times r}$. The affine and $\mathcal{C}$ constraints are lifted to the objective via an indicator function:

$${\delta }_{\mathcal{C}}\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}x\in \mathcal{C},\hfill \\ \infty \hfill & \mathrm{else}.\hfill \end{array}\right.$$

The notation $A_{\mathsf{\Omega }}$ for a symmetric matrix A is the projection of A on the sparsity pattern $\mathsf{\Omega }$:

$${\left(A_{\mathsf{\Omega }}\right)}_{ij}=\left\{\begin{array}{cc}{A}_{ij},\hfill & \mathrm{if}\{i,j\}\in \mathsf{\Omega }\hfill \\ 0,\hfill & \mathrm{else},\hfill \end{array}\right.$$

and we write $A\in {\mathbb{S}}_{\mathsf{\Omega }}^n$ if $A_{\mathsf{\Omega }}=A$. Specifically, $\mathsf{\Omega }$ captures the effective sparsity of the problem; that is, $f\left(Z\right)=f\left(Z_{\mathsf{\Omega }}\right)$ and $\mathcal{A}\left(Z\right)=\mathcal{A}\left(Z_{\mathsf{\Omega }}\right)$. We assume $\{i,i\}\in \mathsf{\Omega }$ for all i, so the second is trivially true.

4.1. Duality

As shown in [69], a notion of a dual problem can be established via the augmented Lagrangian of (11):

$$\begin{array}{ccc}\hfill & & {\mathcal{L}}_{\rho }(Z,X,Y;S,U)=\hfill \\ & & f\left(Z\right)+{\delta }_{\mathcal{C}}\left(Y\right)+\langle U,X-Y\rangle +\langle S,Z-X{Y}^T\rangle \hfill \\ & & +\frac{\rho }{2}{\parallel X-Y\parallel }_F^2+\frac{\rho }{2}{\parallel Z-X{Y}^T\parallel }_F^2,\hfill \end{array}$$

(12)

where the dual problem is:

$$\underset{S,U}{\max }\underset{Z,X,Y}{min}{\mathcal{L}}_{\rho }(Z,X,Y;S,U).$$

The minimization of ${\mathcal{L}}_{\rho }$ over Z and X is the solution to:

$$\begin{array}{ccc}\hfill \nabla f\left(Z\right)-{\mathcal{A}}^{*}\left(\nu \right)+S+\rho (Z-X{Y}^T)& =& 0\hfill \\ \hfill U-SY+\rho (X{Y}^{T}Y-ZY)+\rho (X-Y)& =& 0\hfill \\ \hfill \mathcal{A}\left(Z\right)& =& b,\hfill \end{array}$$

(13)

where $\nu >0$ is a Lagrange dual variable for the local constraint $\mathcal{A}\left(Z\right)=b$. The minimization of ${\mathcal{L}}_{\rho }$ over Y is the solution to the generalized projection problem:

$$\underset{Y\in \mathcal{C}}{min}{\langle Y-\widehat{Y},Y-\widehat{Y}\rangle }_H=\mathbf{Tr}\left((Y-\widehat{Y})H{(Y-\widehat{Y})}^T\right),$$

(14)

where: $\widehat{Y}=U+SX+\rho (X+Z^{T}X),H=\rho (I+X^{T}X).$ For general nonconvex problems, it is difficult to guarantee global minimality. Here, we introduce two sought-after properties that are more reasonably attainable.

 Definition 1 

([70]). The tangent cone of a nonconvex set $\mathcal{C}$ at x is given by ${\mathcal{T}}_{\mathcal{C}}\left(x\right)=\{d:\mathrm{for}\mathrm{all}t\to 0,\widehat{x}\to x,\widehat{x}\in \mathcal{C},\mathrm{there}\mathrm{exists}\widehat{d}\to d,\widehat{x}+t\widehat{d}\in \mathcal{C}\}.$ The normal cone of $\mathcal{C}$ at x ($:={\mathcal{N}}_{\mathcal{C}}\left(x\right)$) is the polar of the tangent cone.

 Definition 2. 

For a minimization of a smooth constrained function $\underset{x\in \mathcal{C}}{min}f\left(x\right)$ we say that $x^{*}$ is a KKT-stationary point if $-\nabla f\left(x^{*}\right)\in {\mathcal{N}}_{\mathcal{C}}\left(x^{*}\right)$.

 Definition 3. 

For a function defined over M variables $\mathcal{L}(X_1,\cdots ,X_m)$, we say that $X_1^{*},\dots ,X_m^{*}$ are (block) coordinatewise minimum points if for each $k=1,\cdots ,m$,

$X_k^{*}=\underset{X}{\mathrm{argmin}}\mathcal{L}(X_1^{*},\dots ,X_{k-1}^{*},X,X_{k+1}^{*},\cdots ,X_m^{*}).$

Note that it is not always the case that stationarity is stronger than coordinatewise minimum. A simple example is $\mathcal{C}={\{-1,1\}}^n$. Then, for all points $x\in \mathcal{C}$, the tangent cone is $\left\{0\right\}$ and the normal cone is ${\mathbb{R}}^n$. Then, every point in $\mathcal{C}$ is stationary, no matter the objective function.

 Proposition 1. 

If Algorithm 1 converges to coordinatewise minimum points $({(X,Z)}^{*},Y^{*},S^{*},U^{*})$, then the primal points (i) satisfy (13) for some choice of $\nu \ge 0$, (ii) minimize (14), (iii) and are primal-feasible, e.g., $X^{*}=Y^{*}$ and ${\left(X^{*}{\left(Y^T\right)}^{*}\right)}_{\mathsf{\Omega }}=Z^{*}$. Furthermore, $(X^{*},Y^{*},Z^{*},S^{*},U^{*})$ are stationary points of (12).

 Proof. 

It is clear that the convergent points of Algorithm 1 exactly satisfy the three conditions. To show that these points are stationary, note that the augmented Lagrangian is convex with respect to $X,Z$ jointly, and is a projection on a compact set $\mathcal{C}$ with respect to Y. Therefore:

$$\begin{array}{c}\hfill {\nabla }_{X,Z,S,U}{\mathcal{L}}_{\rho }(Z^{*},X^{*},Y^{*};S^{*},U^{*})=0,\\ \hfill -{\nabla }_Y{\overline{\mathcal{L}}}_{\rho }(Z^{*},X^{*},Y^{*};S^{*},U^{*})\in {\mathcal{N}}_{\mathcal{C}}\left(Y^{*}\right),\end{array}$$

(15)

where ${\overline{\mathcal{L}}}_{\rho }(Z,X,Y;S,U)=-\langle U,Y\rangle -\langle S,X{Y}^T\rangle +\frac{\rho }{2}{\parallel X-Y\parallel }_F^2+\frac{\rho }{2}{\parallel Z-X{Y}^T\parallel }_F^2,$ with all the differentiable terms of ${\mathcal{L}}_{\rho }$ involving Y.    □

4.2. Linearized ADMM

We propose to solve (11) via the linearized ADMM, e.g., where at each iteration, the objective is replaced by its current linearization:

$$f\left(Z\right)\approx {\widehat{f}}^k\left(Z\right):=f\left(Z^{k-1}\right)+\langle \nabla f\left(Z^{k-1}\right),Z-Z^{k-1}\rangle .$$

We then build the linearized augmented Lagrangian function as:

$$\begin{array}{c}\hfill {\widehat{\mathcal{L}}}^k(Z,X,Y;S,U)=g_k(X,Z)+h\left(Y\right)+\langle U,X-Y\rangle \\ \hfill +\langle S,Z-X{Y}^T\rangle +\frac{\rho }{2}{\parallel X-Y\parallel }_F^2+\frac{\rho }{2}{\parallel Z-X{Y}^T\parallel }_F^2\end{array}$$

(16)

where $g_k(X,Z)={\widehat{f}}^k\left(Z\right)+{\delta }_{\left\{0\right\}}(\mathcal{A}\left(Z\right)-b),h\left(Y\right)={\delta }_{\mathcal{C}}\left(Y\right)$ and $S\in {\mathbb{R}}^{n\times n}$ and $U\in {\mathbb{R}}^{n\times r}$ are the dual variables corresponding to the two coupling constraints. The full algorithm is given in Algorithm 1.

Algorithm 1 ADMM for solving (11)

1: Inputs: ${\rho }_0>0$, $\alpha >1$, tol $ϵ>0$ 2: Initialize: $Z^0,X^0;S^0,U^0$ as random matrices 3: Outputs: Z, $X=Y$ 4: for $k=1\cdots$do 5:     Update $Y^{k+1}$ the solution of: $$\begin{array}{c}\underset{Y\in {\mathbb{R}}^{n\times k}}{\min }\parallel Z^k-X^k{Y}^T+\frac{S^k}{{\rho }^k}{\parallel }_F^2+{\parallel X^k-Y+\frac{U^k}{{\rho }^k}\parallel }_F^2,\hfill \\ s.t.Y\in \mathcal{C}\hfill \end{array}$$ (17) 6:     Update ${(Z,X)}^{k+1}$ as the solutions of: $$\begin{array}{c}\hfill \underset{X,Z\in {\mathbb{S}}_{\mathsf{\Omega }}^n}{min}{\mathcal{L}}_{k+1}(Z,X,Y^{k+1};S^k,U^k;{\rho }^k),\\ \hfill s.t.\mathcal{A}\left(Z\right)=b\end{array}$$ (18) where $\mathcal{L}$ is the linearized augmented Lagrangian as defined in (16). 7:     Update $S,U$ and $\rho$ via: $$\begin{array}{cc}{S}^{k+1}=\hfill & S^k+{\rho }^k{(Z^{k+1}-X^{k+1}{\left(Y^{k+1}\right)}^T)}_{\mathsf{\Omega }}\hfill \\ U^{k+1}=\hfill & U^k+{\rho }^k(X^{k+1}-Y^{k+1})\hfill \\ {\rho }^{k+1}=\hfill & \alpha {\rho }^k\hfill \end{array}$$ (19) 8:     if $max\{\parallel X^k-Y^k\parallel ,\parallel {(Z^k-X^k{\left(Y^k\right)}^T)}_{\mathsf{\Omega }}\parallel \}\le ϵ$ then 9:           break 10:      end if 11: end for

4.2.1. Minimizing over Y

The generalized projection (14) can be solved a number of ways. Note that if $r=1$, then H is a positive scalar, and the problem reduces to $Y^{+}={\mathbf{proj}}_{Y\in \mathcal{C}}\left(\frac{1}{H}\widehat{Y}\right)$. When $\mathcal{C}={\{-1,1\}}^n$, this process reduces to recovering the signs of $\widehat{Y}$ i.e., $Y_i={\mathbf{sign}}_{\mathcal{C}}\left({\widehat{Y}}_i\right)$, and when $\mathcal{C}=\{u:\parallel u{\parallel }_2=1\}$ the set of unit-norm vectors, Y is just a properly scaled version of $\widehat{Y}$: $Y=\frac{1}{\parallel \widehat{Y}{\parallel }_2}\widehat{Y}.$ However, in general, it is difficult to compute the generalized projection over a nonconvex set. When $\mathcal{C}$ is convex, the generalized projection problem (14) can be computed using projected gradient descent. Note that the objective of (14) is 1-strongly convex; thus, we expect fast convergence in this subproblem. In practice, we find that if r is not too large, often a few tens of iterations is enough.

4.2.2. Minimizing over X and Z

Using standard linear algebra techniques, the linear system (13) can be reduced to a few simple instructions. First, we solve for the Lagrange dual variable $\nu$ associated with the linear constraints (and localized to the minimization of X and Z):

$$\begin{array}{ccc}\hfill & & \mathcal{A}\left({\mathcal{A}}^{*}\left(\nu \right)(Y{Y}^T+I)\right)=\hfill \\ & & \rho (b-\mathcal{A}(D{Y}^T+Y{Y}^T))+\mathcal{A}\left((G+S)(I+Y{Y}^T)\right),\hfill \end{array}$$

(20)

where $D=\frac{1}{\rho }(SY-U)+Y$ and $G=\nabla f\left(Z^{k-1}\right)$ the local gradient estimate. When $\mathcal{A}=\mathbf{diag}$, (20) reduces to n scalar element-wise computations ${\nu }_i=\frac{\rho (b-{\left(D{Y}^T\right)}_{ii})+{\left((G+S)(I+Y{Y}^T)\right)}_{ii}}{{\left(Y{Y}^T\right)}_{ii}+1}.$ When $\mathcal{A}=\mathbf{Tr}$, $\nu =\frac{\rho (b-\mathbf{Tr}\left(D{Y}^T\right)+\mathbf{Tr}\left((G+S)(I+Y{Y}^T)\right)}{\mathbf{Tr}\left(Y{Y}^T\right)+1}$. Note that in both cases, no $n\times n$ matrix need ever be formed, so the memory requirement remains $O\left(nr\right)$. (See Appendix A for elaboration). Then, the primal variables are recovered via $X=BY+D,\mathrm{and}Z={\left(X{Y}^T\right)}_{\mathsf{\Omega }}+B,$ with $B=-\frac{1}{\rho }(C-{\mathcal{A}}^{*}\left(\nu \right)+S).$ In these cases, the complexity is dominated by multiplications between $n\times n$ and $n\times r$ matrices. Thus, the method is especially efficient when $r\ll n$.

4.3. Convergence Analysis

 Theorem 1. 

Assume that $f\left(Z\right)$ is $L_f$-smooth. Assume the dual variables are bounded, e.g., $max\{\parallel S^k{\parallel }_F,\parallel U^k{\parallel }_F,\parallel Y^k{{\parallel }_F\}}_k\le B_P<+\infty ,$ and $\frac{L_f}{{\sigma }_{max}}$ is bounded above, where ${\sigma }_{max}=1-\frac{\sqrt{{\sigma }_Y^4+4{\sigma }_Y^2}-{\sigma }_Y^2}{2},{\sigma }_Y={\parallel Y^{k+1}\parallel }_2.$ Then, by running Algorithm 1 with ${\rho }^k=\alpha {\rho }^{k-1}={\alpha }^k{\rho }_0$, if ${\mathcal{L}}_k$ is bounded below, then the sequence $\{P^k,D^k\}$ converges to a stationary point of (12).

 Proof. 

See Appendix B.    □

 Corollary 1. 

If $r\ge ⌈\sqrt{2n}⌉$ and the stationary point of Algorithm 1 converges to a second-order critical point of (1), then it is globally optimal for the convex relaxation of (10) [13].

Unfortunately, the extension of KKT stationary points to global minima is not yet known when $\frac{r(r+1)}{2}<n$ (i.e., $r=1$). However, our empirical results suggest that even when $r=1$, often a local solution to (10) well-approximates the global solution to (1).

5. ADMM on Simplified Nonconvex SDP

When the linear constraints are not present, (1) can be reformulated without Z, into:

$$\underset{X,Y}{\min }g\left(X\right)+{\delta }_{\mathcal{C}}\left(Y\right),s.t.X=Y,$$

(21)

with matrix variables $X\in {\mathbb{R}}^{n\times r},Y\in {\mathbb{R}}^{n\times r}$, and where $g\left(X\right)=f\left(X{X}^T\right)$ is smooth. We can also define an augmented Lagrangian of (21) as ${\mathcal{L}}_{\rho }(X,Y;U)=g\left(X\right)+{\delta }_{\mathcal{C}}\left(Y\right)+\langle U,X-Y\rangle +\frac{\rho }{2}{\parallel X-Y\parallel }_F^2.$

 Theorem 2. 

The coordinatewise minimum points $X^{*}=Y^{*}$ satisfying:

$$\begin{array}{ccc}\hfill 0& =& \nabla g\left(X^{*}\right)+U+\rho (X-Y)\hfill \\ \hfill Y& =& {\displaystyle {\mathbf{proj}}_{\mathcal{C}}(X+\frac{1}{\rho }U)}\hfill \\ \hfill X& =& Y,\hfill \end{array}$$

(22)

are the stationary points of the problem:

$$\underset{X}{\min }g\left(X\right),s.t.X\in \mathcal{C}.$$

(23)

 Proof. 

The KKT stationary points of (23) can be characterized in terms of the normal cone of $\mathcal{C}$ at $X^{*}$; specifically, $X^{*}$ is stationary if:

$$\langle \nabla g\left(X^{*}\right),X-X^{*}\rangle \le 0,\forall X\in \mathcal{C}\cap {\mathcal{N}}_{ϵ}\left(X^{*}\right),$$

where ${\mathcal{N}}_{ϵ}\left(X^{*}\right)$ is some small neighborhood containing $X^{*}$. (This is an equivalent definition of the Clarke stationary point [70], since in a close enough neighborhood to $X^{*}$, the subdifferential of ${\delta }_{\mathcal{C}}\left(x\right)$ is ${\mathcal{N}}_{\mathcal{C}}\left(x\right)$).

Combining terms in (22) gives $X^{*}=Y^{*}$ satisfying $X^{*}={\mathbf{proj}}_{\mathcal{C}}\left(X^{*}-\frac{1}{\rho }\nabla g\left(X^{*}\right)\right).$ The optimality condition of the projection is $\langle X-(X-\frac{1}{\rho }\nabla g\left(X^{*}\right)),X-X^{*}\rangle \le 0,\forall X\in \mathcal{C}\cap {\mathcal{N}}_{ϵ}\left(X^{*}\right)$ which reduces to the desired condition.    □

5.1. ADMM

The alternating steps in minimizing the augmented Lagrangian over the primal variables are extremely simple, compared with the previous matrix formulation. In general, we are considering $f\left(X\right)$ linear (in which case the update of X involves only addition) or quadratic with strictly positive diagonal Hessian (which adds a small scaling step). $\mathcal{C}={\{-1,1\}}^n,\mathcal{C}={\{x:\parallel x\parallel }_2=1\},$ even when $r>1$.

5.2. Convergence Analysis

 Definition 4. 

A differentiable convex function $g\left(X\right)$ is $L_g$-smooth and $H_g$-strongly convex over ${\mathbb{R}}^n$ if for any X, Y, $g\left(X\right)-g\left(Y\right)\ge \langle \nabla f\left(X\right),X-Y\rangle -\frac{L_g}{2}{\parallel X-Y\parallel }_F^2$ and $g\left(X\right)-g\left(Y\right)\le \langle \nabla f\left(X\right),X-Y\rangle -\frac{H_g}{2}{\parallel X-Y\parallel }_F^2.$

 Theorem 3. 

Assume $g\left(X\right)$ is lower bounded over $\mathcal{C}$, and is $L_g$-smooth. Given a sequence $\left\{{\rho }^k\right\}$ such that:

$$\frac{{\rho }^k-3L_g}{2}-L_g^2\frac{{\rho }^{k+1}+{\rho }^k}{2{\left({\rho }^k\right)}^2}>0,{\rho }^k>L_g$$

for all k, then under Algorithm 2 the augmented Lagrangian $\mathcal{L}(X^k,Y^k;U^k)$ is lower bounded and convergent, with $\{X^k,Y^k,U^k\}\to \{X^{*},Y^{*},U^{*}\}$ a stationary and feasible solution of (23).

 Proof. 

See Appendix C.    □

 Remark 1. 

Convergence is guaranteed under a constant penalty coefficient ${\rho }_k\equiv {\rho }^0\ge \frac{3+\sqrt{17}}{2}{L}_g,\alpha =1.$ However, in implementation, we find empirically that increasing $\left\{{\rho }^k\right\}$ from a relatively small ${\rho }^0$ can encourage convergence to more useful global minima.

Algorithm 2 ADMM for solving (23)

1: Inputs: ${\rho }_0>0$, $\alpha >1$, tol $ϵ>0$ 2: Initialize: $Z^0,X^0;S^0,U^0$ as random matrices 3: Outputs: Z, $X=Y$ 4: for $k=1\cdots$do 5:     Update $Y^{k+1}$ the solution of: $$\underset{Y\in R^{n\times k}}{\min }{\parallel X^k-Y+\frac{U^k}{{\rho }^k}\parallel }_F^2,s.t.Y\in \mathcal{C}.$$ (24) 6:     Update $X^{k+1}$ as the solution of: $$0=\nabla g\left(X\right)+U+\rho (X-Y).$$ (25) 7:     Update U and $\rho$ via: $$\begin{array}{cc}{U}^{k+1}=\hfill & U^k+{\rho }^k(X^{k+1}-Y^{k+1}),\hfill \\ {\rho }^{k+1}=\hfill & \alpha {\rho }^k.\hfill \end{array}$$ (26) 8:     if $\parallel X^k-Y^k{\parallel }_F \le ϵ$ then 9:          break 10:      end if 11: end for

 Theorem 4. 

If $g\left(X\right)$ is $H_g$-strongly convex and ${\rho }^k=\rho$ constant, with $\frac{\rho +H_g}{2}\ge \frac{L_g^2}{\rho },\rho >L_g$ then under Algorithm 2 the augmented Lagrangian $\mathcal{L}(X^k,Y^k;U^k)$ converges to $\mathcal{L}(X^{*},Y^{*},U^{*})$ at a linear rate.

 Proof. 

See Appendix C.1. □

6. Numerical Experiments

In this section, we give numerical results on the proposed methods for community detection, MAX-CUT, image segmentation, and symmetric matrix factorization. In each application, we evaluate and compare these four methods. (i) SD: the solution to a semidefinite relaxation of (1) (SDR), where $\mathcal{C}={\mathbb{R}}^{n,r}$. The binary vector factor x where $x{x}^T=Z$ is recovered using a Goemans–Williamson style rounding. [9] technique. This is our baseline method and is described in more detail below. (ii) MR1: Algorithm 1 with $r=1$. (iii) MRR: Algorithm 1 with $r=⌈\sqrt{2n}⌉$, then rounded to a binary vector using a nonsymmetric version of the Goemans–Williamson style rounding [9] technique. Both MR1 and MRR have the following stopping criterion $max\{P^{\left(k\right)},D^{\left(k\right)}\}\le ϵ$ for some tolerance parameter $ϵ>0$, where: $P^{\left(k\right)}:=\left\{\frac{\parallel Z^k-Z^{k-1}{\parallel }_2}{\parallel Z^k{\parallel }_2},\frac{\parallel X^k-X^{k-1}{\parallel }_2}{\parallel X^k{\parallel }_2},\frac{\parallel Y^k-Y^{k-1}{\parallel }_2}{\parallel Y^k{\parallel }_2}\right\},$$D^{\left(k\right)}:=max\left\{\frac{\parallel Z^{\left(k\right)}-X^{\left(k\right)}{\left(Y^{\left(k\right)}\right)}^T{\parallel }_2}{\parallel Z^k{\parallel }_2},\frac{\parallel X^{\left(k\right)}-Y^{\left(k\right)}{\parallel }_2}{\parallel X^{\left(k\right)}{\parallel }_2}\right\}.$ (Here, $D^{\left(k\right)}$ is also proportional to the difference in dual iterates, and thus $P^{\left(k\right)}$ and $D^{\left(k\right)}$ can be interpreted as primal and dual residuals, respectively). (iv) V: Algorithm 2, with stopping criterion $max\{P^{\left(k\right)},D^{\left(k\right)}\}\le ϵ$ where $P^{\left(k\right)}:=\left\{\frac{\parallel x^k-x^{k-1}{\parallel }_2}{\parallel x^k{\parallel }_2},\frac{\parallel y^k-y^{k-1}{\parallel }_2}{\parallel y^k{\parallel }_2}\right\},D^{\left(k\right)}:=\frac{\parallel x^k-y^k{\parallel }_2}{\parallel x^k{\parallel }_2}.$ The same primal and dual residual interpretation can be used here as well. In all cases, we use the following scheme for $\rho$: ${\rho }^k=min\{{\rho }_{max},{\rho }^{k-1}*\gamma \},$ where ${\rho }_{max}\approx 10,000$ and $\gamma \approx 1.05$ (slightly larger than 1).

6.1. Solving the Baseline (SDR)

As a baseline, we compare against the solution of the semidefinite relaxed problem without factor variables X (e.g., $\mathcal{C}={\mathbb{R}}^{n,n}$):

$$\underset{Z}{\min }f\left(Z\right),s.t.\mathcal{A}\left(Z\right)=b,Z⪰0.$$

(27)

For a fair comparison, we use a first-order splitting method very similar to ADMM, which is the Douglas–Rachford Splitting (DRS) method ([71,72], see also [73,74]). We introduce dummy variables and solve the reformulation of (27):

$$\underset{Z_1,Z_2,Z_3}{\min }{g}_1\left(Z_1\right)+g_2\left(Z_2\right)+g_3\left(Z_3\right),s.t.Z_1+Z_2+Z_3,$$

where $g_1\left(Z_1\right)=\mathbf{Tr}\left(C{Z}_1\right),g_2\left(Z_2\right)=\left\{\begin{array}{cc}0,\hfill & \mathcal{A}\left(Z_2\right)=b\hfill \\ +\infty ,\hfill & \mathrm{else},\hfill \end{array}\right.g_3\left(Z_3\right)=\left\{\begin{array}{cc}0,\hfill & Z_3⪰0\hfill \\ +\infty ,\hfill & \mathrm{else}.\hfill \end{array}\right.$. An application of the DRS on this reformulation (see also Algorithm 3.1 in [75]) is then the following iteration scheme: for $i=1,2,3$,

$$\begin{array}{ccc}\hfill X_i^{(k+1)}& =& {\mathbf{prox}}_{t{g}_i}\left(Z_i\right),{\widehat{Y}}_i=2X_i^{(k+1)}-Z_i^{\left(k\right)},\hfill \\ \hfill Y^{(k+1)}& =& \frac{1}{3}(X_1^{(k+1)}+X_2^{(k+1)}+X_3^{(k+1)}),\hfill \\ \hfill Z_i^{(k+1)}& =& Z_i^{\left(k\right)}+\rho (Y^{(k+1)}-X_i^{(k+1)})\hfill \end{array}$$

and for a convex function f $z={\mathbf{prox}}_{tf}\left(u\right)\iff \underset{z}{\mathrm{argmin}}f\left(z\right)+\frac{1}{2t}{\parallel z-u\parallel }_2^2.$

6.2. Rounding

Following the technique in [9], we can estimate x from a rank r matrix $X\approx x{x}^T$ by randomly projecting the main eigenspaces on the unit sphere. The exact procedure is as follows. (i) For the symmetric SDP solution X, we first perform an eigenvalue decomposition $X=Q\Lambda Q^T$ and form a factor $F=Q{\Lambda }^{1/2}$ where the diagonal elements of $\Lambda$ are in decreasing magnitude order. Then, we scan $k=1,\cdots ,n$ and find $x_{k,t}=\mathbf{sign}\left(F_k{z}_t\right)$ for trials $t=1,\cdots 10$. Here, $F_k$ contains the first k columns of F, and each element of $z_t\in {\mathbb{R}}^k$ is drawn i.i.d from a normal Gaussian distribution. We report the values for $x_r=\underset{x_{k,t}}{\mathrm{argmin}}\left\{x_r^{T}C{x}_r\right\}$. (ii) For the MRR method, we repeat the procedure using a factor $F=U{\Sigma }^{1/2}$ where $X=U\Sigma V^T$ is the SVD of X. (iii) For MR1 and V, we simply take $x_r=\mathbf{sign}\left(x\right)$ as the binary solution.

6.3. Computer Information

The following simulations are performed on a standard desktop computer with an Intel Xeon processor (3.6 GHz), and 32 GB of RAM. It is running with Matlab R2017a.

6.4. MAX-CUT

Table 1. MAX-CUT values for graphs from the 7th DIMACS Challenge. MRR = matrix formulation, $r=⌈\sqrt{2n}⌉$. SDR = SDP relaxation + rounding technique.

Database	n	Sparsity	BK	V	MR1	MRR	SDR
g3-8	512	0.012	41,684,814	34,105,231	36,780,180	35,943,350	33424095
g3-15	3375	0.018	281,029,888	235,893,612	255,681,256	241,740,931	212,669,181
pm3-8-50	512	0.012	454	394	346	378	416
pm3-15-50	3375	0.018	2964	2594	1966	2140	2616
G1	800	0.0599	11,624	10,938	11,047	11,321	11,360
G2	800	0.0599	11,620	10,834	11,082	11,144	11,343
G3	800	0.0599	11,622	10,858	10,894	11,174	11,367
G4	800	0.0599	11,646	10,849	10,760	11,192	11,429
G5	800	0.0599	11,631	10,796	10,783	11,352	11,394
G6	800	0.0599	2178	1853	1820	1949	1941
G7	800	0.0599	2003	1694	1644	1705	1774
G8	800	0.0599	2003	1688	1641	1728	1766
G9	800	0.0599	2048	1771	1681	1807	1830
G10	800	0.0599	1994	1662	1641	1737	1732
G11	800	0.005	564	496	460	480	506
G12	800	0.005	556	486	448	480	512
G13	800	0.005	580	516	476	498	528
G14	800	0.0147	3060	2715	2768	2861	2901
G15	800	0.0146	3049	2625	2810	2803	2884
G16	800	0.0146	3045	2667	2736	2862	2910
G17	800	0.0146	3043	2638	2789	2840	2920
G18	800	0.0147	988	798	768	841	858
G19	800	0.0146	903	700	641	694	780
G20	800	0.0146	941	723	691	766	788
G21	800	0.0146	931	696	713	810	794
G22	2000	0.01	13,346	12,461	12,548	12,751	12,926
G23	2000	0.01	13,317	12,540	12,528	12,853	12,889
G24	2000	0.01	13,314	12,540	12,447	12,723	12,904
G25	2000	0.01	13,326	12,447	12,558	12,733	12,874
G26	2000	0.01	13,314	12,445	12,475	12,718	12,847
G27	2000	0.01	3318	2824	2508	2807	2909
G28	2000	0.01	3285	2753	2518	2796	2845
G29	2000	0.01	3389	2864	2628	2901	2896
G30	2000	0.01	3403	2887	2639	2937	2971
G31	2000	0.01	3288	2833	2518	2902	2825
G32	2000	0.002	1398	1220	1066	1204	1254
G33	2000	0.002	1376	1202	1054	1166	1250
G34	2000	0.002	1372	1208	1096	1170	1222
G35	2000	0.0059	7670	6605	6914	6764	7209
G36	2000	0.0059	7660	6564	6943	6598	7228
G37	2000	0.0059	7666	6478	6839	6789	7183
G38	2000	0.0059	7681	6486	6759	6768	7212
G39	2000	0.0059	2395	1616	1697	1840	1997
G40	2000	0.0059	2387	1617	1438	1921	1890
G41	2000	0.0059	2398	1606	1656	1778	1899
G42	2000	0.0059	2469	1707	1756	1862	1971
G43	1000	0.02	6659	6222	6236	6398	6475
G44	1000	0.02	6648	6275	6192	6447	6458
G45	1000	0.02	6652	6243	6255	6407	6454
G46	1000	0.02	6645	6217	6233	6398	6407
G47	1000	0.02	6656	6221	6266	6433	6454
G48	3000	0.0013	6000	5882	5006	5402	6000
G49	3000	0.0013	6000	5844	5038	5362	6000
G50	3000	0.0013	5880	5814	4994	5410	5880
G51	1000	0.0118	3846	3317	3446	3524	3642
G52	1000	0.0118	3849	3360	3471	3499	3662
G53	1000	0.0118	3846	3323	3510	3516	3660
G54	1000	0.0118	3846	3306	3428	3509	3651

gives the best MAX-CUT values using best-of-random-guesses and our approaches over four examples from the seventh DIMACS Implementation Challenge in 2002 (see http://dimacs.rutgers.edu/Workshops/7thchallenge/, problems downloaded from http://www.optsicom.es/maxcut/). Often, we find the quality of our recovered solutions close to the best-known solutions and often achieve similar suboptimality as the rounded SDR solutions. However, the runtime comparison (Figure 1) suggests that the ADMM methods (especially MR1 and SDR) are much more computationally efficient and scalable. All experiments are performed with $ϵ=1\times {10}^{-3}$.

6.5. Image Segmentation

Both community detection and MAX-CUT can be used in image segmentation, where each pixel is a node and the similarity between pixels forms the weight of the edges. Generally, solving (1) for this application is not preferred, since the number of pixels in even a moderately sized image is extremely large. However, because of our fast methods, we successfully performed image segmentation on several thumbnail-sized images, as seen in Figure 2.

The C matrix is composed as follows. For each pixel, we compose two feature vectors: $f_c^{ij}$ containing the RGB values and $f_p^{ij}$ containing the pixel location. Scaling $f_c^{ij}$ by some weight c, we form the concatenated feature vector $f^{ij}=[f_c^{ij},c{f}_p^{ij}]$, and form the weighted adjacency matrix as the squared distance matrix between each feature vector $A_{\left(ij\right),\left(kl\right)}={\parallel f^{ij}-f^{kl}\parallel }_2^2$. For MAX-CUT, we again form $C=A-\mathbf{Diag}\left(A\mathbf{1}\right)$ as before. For community detection, since we do not have exact p and q values, we use an approximation as $C=a\mathbf{1}{\mathbf{1}}^T-A$ where $a=\frac{1}{n^2}{\mathbf{1}}^{T}A\mathbf{1}$ the mean value of A. Sweeping C and ${\rho }_0$, we give the best qualitative result in Figure 2.

6.6. Symmetric Factorization with Partial Observations

Recall the factorization with partial observations formulation as follows:

$${\displaystyle \underset{Z\in {\mathbb{S}}^n,X\in {\mathbb{R}}^{n\times r}}{\min }\sum _{i,j\in \mathsf{\Omega }}{(Z_{ij}-C_{ij})}^2,s.t.Z=X{X}^T,X\ge 0.}$$

(28)

Note that here we generalize the aforementioned formulation with $r=5$. In this setting, while the strongly convex $Y—$update in the proposed algorithm can no longer be solved in closed form, projected gradient descent is applied to deal with it. The relative error defined as $\parallel {(Z^{*}-C)}_{\mathsf{\Omega }}\parallel /\parallel C_{\mathsf{\Omega }}\parallel$ and CPU time with varying problem size and sparsity are demonstrated in

Table 2. Result for nonnegative factorization with partial observations from linearized ADMM (5 trials). STD = standard deviation.

n	1000			3000			5000			8000
$\left|\mathsf{\Omega }\right|/n^2$	0.1	0.5	0.8	0.1	0.5	0.8	0.1	0.5	0.8	0.1	0.5	0.8
CPU time/s	9.74	13.53	13.97	61.15	78.99	64.76	117.54	85.24	131.64	212.26	220.42	337.74
$\frac{\parallel {(Z^{*}-C)}_{\mathsf{\Omega }}\parallel }{\parallel C_{\mathsf{\Omega }}\parallel }$	0.86	0.85	0.86	0.89	0.89	0.89	0.89	0.88	0.87	0.88	0.90	0.89
STD	0.043	0.020	0.021	0.010	0.006	0.008	0.008	0.012	0.018	0.004	0.008	0.008

.

7. Conclusions

We present two methods for solving quadratic combinatorial problems using ADMM on two reformulations. Though the problem has a nonconvex constraint, we give convergence results to KKT solutions under mild conditions. From this, we give empirical solutions to several graph-based combinatorial problems, specifically MAX-CUT and community detection; both can be used in additional downstream applications, like image segmentation.