Near-Optimal Graph Signal Sampling by Pareto Optimization

Abstract

In this paper, we focus on the bandlimited graph signal sampling problem. To sample graph signals, we need to find small-sized subset of nodes with the minimal optimal reconstruction error. We formulate this problem as a subset selection problem, and propose an efficient Pareto Optimization for Graph Signal Sampling (POGSS) algorithm. Since the evaluation of the objective function is very time-consuming, a novel acceleration algorithm is proposed in this paper as well, which accelerates the evaluation of any solution. Theoretical analysis shows that POGSS finds the desired solution in quadratic time while guaranteeing nearly the best known approximation bound. Empirical studies on both Erdos-Renyi graphs and Gaussian graphs demonstrate that our method outperforms the state-of-the-art greedy algorithms.

1. Introduction

Graph Signal Processing (GSP) is a powerful tool for irregular data modeling and processing, and has attracted extensive attention from the signal processing [1,2], machine learning [3], and computer vision communities [4,5]. Traditional signal processing usually treat the data as a sequence of vectors, then analyze the features in the time domain or frequency domain, and try to extract information from the features. This implies that the data rely on an Euclidean space ${\mathbb{R}}^n$. However, in many real-world applications such as computer graphics [6] and machine learning [7], this assumption is not accurate enough. For example, the data captured by sensors in a sensor network are always interconnected to each other, and they may rely on a manifold embedded in an Euclidean space. Moreover, a RGB image lies within the pixel space ${\mathbb{R}}^{H\times W\times 3}$, but the dimension of the natural image space is usually much lower. GSP brings the signal from the time (spatial) domain to the node domain, and allows us to use a similar tool in traditional signal processing, such as Fourier transform and wavelet analysis, to analyze the graph signals, and then extract the latent information. As an illustration, Figure 1 shows the graph signals supported on the Stanford bunny graph, the Swiss roll graph and the Minnesota road graph respectively.

However, in some applications, the underlying graph is very large, which makes it very expensive to collect and process the data from all nodes. Graph signal sampling is a basic problem in GSP that aims to select a subset with the minimal reconstruction error from the graph node set. Hence, we just need to collect the signal from a small-sized sampling set, and the complete graph signal can be obtained via optimal graph signal reconstruction. Graph signal sampling has been widely used in many fields, e.g., sensor placement [8] and 3D point-cloud sampling [9]. Previous works [10,11] first considered the graph signal reconstruction problem, which aims to recover the whole graph signal from partial observations, then formulated the graph signal sampling problem as a subset of nodes with the minimal optimal reconstruction error. This sampling problem can be regarded as a subset selection problem, which plays a fundamental role in many fields, such as sparse reciprocal graphical model estimation [12] and sparse plus low rank graphical model estimation [13]. This problem is proved to be generally NP-hard [14], so we may not find a global optimal solution of this problem in polynomial time. A sub-optimal method is the greedy algorithm, which selects an element with the greatest improvement to the objective function at each iteration. The greedy algorithm is very simple, but also very effective. In [15], it is proved that the greedy algorithm achieves $1-e^{-\gamma }$ approximation ratio for normalized (that means $f(⌀)=0,f\left(\mathcal{V}\right)=1$) monotonic weak submodular objective functions in general subset selection problems, where $\gamma$ is the submodularity ratio of the objective function. For submodular objective functions we have $\gamma \ge 1$, then the greedy algorithm guarantees $1-1/e$ performance, which is the best-so-far theoretical guarantee.

Recently, many Multi-Objective Evolutionary Algorithms (MOEAs) such as the Pareto Optimization for Subset Selection (POSS) algorithm have exhibited better performance than greedy algorithms do [16,17,18]. POSS treats the cardinality constraint as an additional objective function of the problem, then the original problem is transformed to a Bi-objective Optimization Problem (BOP). POSS adopts the evolutionary mutation operation to search for different Pareto optimal solutions of the BOP, and finally select a desired Pareto optimal solution.

Though POSS has been proved to be a powerful tool to solve the subset selection problem, it is infeasible to directly apply it to the graph signal sampling problem, since the evaluation of the optimal reconstruction error is very expensive. In this work, we propose the Pareto Optimization for Graph Signal Sampling (POGSS) based on POSS, and an acceleration algorithm is proposed. By introducing the supermodularity ratio of the objective function, we prove that this method theoretically achieves comparable performance as greedy algorithms do in polynomial time, and show that the supermodularity ratio is bounded as well. Our major contributions can be summarized as follows:

• Inspired by POSS, we formulate the graph signal sampling problem as a subset selection problem, and solve this problem by employing Pareto optimization, which has been proved to be superior to greedy methods.
• To accelerate the objective estimation procedure, we propose a novel acceleration algorithm based on matrix inversion lemma. Our acceleration algorithm is scalable since it can be used to accelerate the evaluation of $any$ solution, thus that can be directly applied to other evolutionary-based methods (e.g., POSS with recombination [17]).
• We provide comprehensive theoretical analyses of the proposed algorithm, including complexity analysis, Pareto optimality analysis and error bound analysis.

The remainder of this paper is organized as follows. We present the preliminaries and formulate the problem in Section 2.1. We give a brief introduction to the POSS method and introduce the Pareto optimization for graph signal sampling (POGSS) algorithm in Section 3. In Section 4, we provide comprehensive theoretical analyses of POGSS. The Numerical results and the conclusions are presented in Section 5 and Section 6, respectively.

2. Preliminaries

2.1. Notations

In this paper, we use the bold uppercase letters (e.g., $\mathbf{A}$) to denote matrices and use bold lowercase letters (e.g., $\mathbf{a}$) to denote vectors. $(i,j)$-th entry of $\mathbf{A}$ is denoted as $A_{ij}$ or ${\left[\mathbf{A}\right]}_{ij}$. ${(·)}^T$ denote the transpose, ${(·)}^{-1}$ denote the inverse, ${(·)}^{†}$ denote the Moore-Penrose pseudo-inverse. $Tr(·)$ denote the trace. $\mathbb{E}[·]$ denote the expectation. $diag\left(\mathbf{a}\right)$ stands for a diagonal matrix with the diagonal entries given by a vector $\mathbf{a}$. Sets are denoted by calligraphic letters, e.g., $\mathcal{A}$ (except the set of real number, which is denoted as $\mathbb{R}$). $\left[n\right]$ denote the set of all integers from 1 to n, i.e., $\left[n\right]=\{1,2,\cdots ,n\}$. $\lceil ·\rceil$ denote the ceil operator, e.g., $\lceil 0.1\rceil =1$.

2.2. Graph Signal Processing

We consider a weighted, undirected graph $\mathcal{G}=\{\mathcal{V},\mathcal{E}\}$, where $\mathcal{V}$ is the set of nodes with $\left|\mathcal{V}\right|=n$, $\mathcal{E}$ is the set of edges. Let $\mathbf{A}\in {\mathbb{R}}^{n\times n}$ be the adjacency matrix of $\mathcal{G}$, where $A_{ij}$ is the weight of the edge between nodes i and j. The graph signal $\mathbf{x}$ is a function that maps each node in $\mathcal{V}$ to a real number, and can be written as a vector with its’ i-th entry denotes the signal value at nodes i.

Defined the unnormalized graph Laplacian matrix of $\mathcal{G}$ as

$$\begin{array}{c}\hfill \mathbf{L}=\mathbf{D}-\mathbf{A},\end{array}$$

where $\mathbf{D}$ is a diagonal degree matrix with $D_{ii}={\sum }_{j=1}^n{A}_{ij}$. Then we write the eigenvalue decomposition of the graph Laplacian matrix as

$$\begin{array}{c}\hfill \mathbf{L}=\mathbf{V}\mathsf{\Lambda }{\mathbf{V}}^T,\end{array}$$

where $\mathsf{\Lambda }=\{0={\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_n\}$ is a diagonal matrix of graph frequencies, and $\mathbf{V}$ is the matrix of graph Fourier bases. Then the graph Fourier transform of $\mathbf{x}$ is defined as $\overline{\mathbf{x}}={\mathbf{V}}^T\mathbf{x}$, and the inverse Fourier transform is given by $\mathbf{x}=\mathbf{V}\overline{\mathbf{x}}$. We say a graph signal $\mathbf{x}$ is $\mathcal{K}$-bandlimited, if its’ graph Fourier coefficients indexed by $\mathcal{V}\backslash \mathcal{K}$ are zero, i.e., ${\overline{\mathbf{x}}}_{\mathcal{V}\backslash \mathcal{K}}=\mathbf{0}$. In [19], it was proved that if the sampling set size is not smaller than $\left|\mathcal{K}\right|$, then the $\mathcal{K}$-bandlimited graph signal can be perfectly recovered. However, in many applications we can only observe noise corrupted signals, we cannot find a perfect estimator under noise. Consequently, Bayesian estimation is adopted in the graph signal reconstruction problem.

2.3. Problem Formulation

Let $\mathbf{x}$ be the original noise-free graph signal, and $\mathbf{n}$ be the noise vector. In the graph signal reconstruction problem, only the noise corrupted samples ${\mathbf{y}}_{\mathcal{S}}$ in a sampling set $\mathcal{S}\subseteq \mathcal{V}$ are observable, thus we take the observation model as

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\hfill & \mathbf{y}=\mathbf{x}+\mathbf{n},\hfill \\ \hfill & {\mathbf{y}}_{\mathcal{S}}={\mathbf{C}}_{\mathcal{S}}\mathbf{y},\hfill \end{array}\right.\end{array}$$

(1)

where ${\mathbf{C}}_{\mathcal{S}}\in {\{0,1\}}^{\left|\mathcal{S}\right|\times n}$ is the sampling matrix. We assume that $\mathbf{n}$ is a zero-mean vector with covariance matrix ${\sigma }^2\mathbf{I}$, $\mathbf{x}$ is $\mathcal{K}$-bandlimited and ${\overline{\mathbf{x}}}_{\mathcal{K}}$ is a zero-mean vector with covariance matrix $\overline{\mathsf{\Sigma }}$, hence the covariance matrix of $\mathbf{x}$ is $\mathsf{\Sigma }={\mathbf{V}}_{\mathcal{K}}\overline{\mathsf{\Sigma }}{\mathbf{V}}_{\mathcal{K}}^T$. Graph signal reconstruction is to approximately recover the original graph signal $\mathbf{x}$ from partial noisy observation ${\mathbf{y}}_{\mathcal{S}}$ via a linear transform ${\mathsf{\Phi }}_{\mathcal{S}}\in {\mathbb{R}}^{n\times \left|\mathcal{S}\right|}$, i.e.,

$$\begin{array}{c}\hfill \widehat{\mathbf{x}}\left({\mathsf{\Phi }}_{\mathcal{S}}\right)={\mathsf{\Phi }}_{\mathcal{S}}{\mathbf{y}}_{\mathcal{S}}.\end{array}$$

(2)

The error covariance matrix with respect to ${\mathsf{\Phi }}_{\mathcal{S}}$ is defined as

$$\begin{array}{c}\hfill {\mathbf{K}}_{\mathcal{S}}\left({\mathsf{\Phi }}_{\mathcal{S}}\right)=\mathbb{E}\left[\left(\mathbf{x}-\widehat{\mathbf{x}}\left({\mathsf{\Phi }}_{\mathcal{S}}\right)\right){\left(\mathbf{x}-\widehat{\mathbf{x}}\left({\mathsf{\Phi }}_{\mathcal{S}}\right)\right)}^T\right].\end{array}$$

To find the optimal reconstruction matrix ${\mathsf{\Phi }}_{\mathcal{S}}^{★}$, one needs to minimize the scalar loss function $J\left({\mathsf{\Phi }}_{\mathcal{S}}\right)={\mathbf{z}}^T{\mathsf{\Phi }}_{\mathcal{S}}\mathbf{z}$ for all $\mathbf{z}\in {\mathbb{R}}^n$. The derived optimal recover operator is [10]

$$\begin{array}{c}\hfill {\mathsf{\Phi }}_{\mathcal{S}}^{★}=\mathsf{\Sigma }{\mathbf{C}}^T{\left(\mathbf{C}(\mathsf{\Sigma }+{\sigma }^2\mathbf{I}){\mathbf{C}}^T\right)}^{†},\end{array}$$

(3)

and the correspongding error covariance matrix is given by

$$\begin{array}{c}\hfill {\mathbf{K}}_{\mathcal{S}}^{★}={\mathbf{K}}_{\mathcal{S}}\left({\mathsf{\Phi }}_{\mathcal{S}}^{★}\right)={\mathbf{V}}_{\mathcal{K}}{\left({\overline{\mathsf{\Sigma }}}^{-1}+{\sigma }^{-2}\sum _{i\in \mathcal{S}}{\mathbf{v}}_i{\mathbf{v}}_i^T\right)}^{-1}{\mathbf{V}}_{\mathcal{K}}^T,\end{array}$$

(4)

where ${\mathbf{V}}_{\mathcal{K}}^T=[{\mathbf{v}}_1,\cdots ,{\mathbf{v}}_n]$, and the optimal mean squared error (MSE) is given by

$$\begin{array}{c}\hfill {MSE}_{\mathcal{S}}^{★}=\mathbb{E}\left[\parallel \mathbf{x}-\widehat{\mathbf{x}}\left({\mathsf{\Phi }}_{\mathcal{S}}^{★}\right){\parallel }_2^2\right]=Tr\left({\mathbf{K}}_{\mathcal{S}}^{★}\right).\end{array}$$

(5)

The goal of graph signal sampling is to select a subset $\mathcal{S}\subseteq \mathcal{V}$ with at most k nodes, which minimizes the optimal reconstruction error ${MSE}_{\mathcal{S}}^{★}$, i.e., finding

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathcal{S}}^{★}& ={argmin}_{\mathcal{S}\subseteq \mathcal{V},\left|\mathcal{S}\right|\le k}{MSE}_{\mathcal{S}}^{★}\hfill \\ \hfill & ={argmin}_{\mathcal{S}\subseteq \mathcal{V},\left|\mathcal{S}\right|\le k}Tr\left({\mathbf{K}}_{\mathcal{S}}^{★}\right).\hfill \end{array}\end{array}$$

(6)

Define ${\overline{\mathbf{K}}}_{\mathcal{S}}={\left({\overline{\mathsf{\Sigma }}}^{-1}+{\sigma }^{-2}{\sum }_{i\in \mathcal{S}}{\mathbf{v}}_i{\mathbf{v}}_i^T\right)}^{-1}$, $\mathbf{M}={\mathbf{V}}_{\mathcal{K}}^T{\mathbf{V}}_{\mathcal{K}}$. By using the circular commutation property of trace, (6) is simplified to

$$\begin{array}{c}\hfill {\mathcal{S}}^{★}={argmin}_{\mathcal{S}\subseteq \mathcal{V},\left|\mathcal{S}\right|\le k}Tr\left(\mathbf{M}{\overline{\mathbf{K}}}_{\mathcal{S}}\right)\end{array}$$

(7)

3. Method

In this study, we propose to solve (7) by Pareto optimization. For representable convenience, we use the binary vector $\mathbf{s}\in {\{0,1\}}^n$ to represent the subset $\mathcal{S}\subseteq \mathcal{V}$, the entry $s_i=1$ means that the node $v_i$ is included in $\mathcal{S}$ and $s_i=0$ otherwise. We do not distinguish between the two notations. Please note that the original problem is a constrained problem with single objective function, we now reformulate this problem as an unconstrained BOP:

$$\begin{array}{c}\hfill {argmin}_{\mathbf{s}\in {\{0,1\}}^n}\mathbf{f}\left(\mathbf{s}\right)={\left(f\left(\mathbf{s}\right),c\left(\mathbf{s}\right)\right)}^T,\end{array}$$

(8)

where

$$\begin{array}{cc}\hfill & f\left(\mathbf{s}\right)={MSE}_{\mathcal{S}}^{★}=Tr\left(\mathbf{M}{\overline{\mathbf{K}}}_{\mathcal{S}}\right),\hfill \\ \hfill & c\left(\mathbf{s}\right)=\left\{\begin{array}{cc}{\parallel \mathbf{s}\parallel }_0,\hfill & {\parallel \mathbf{s}\parallel }_0<2k\hfill \\ +\infty ,\hfill & {\parallel \mathbf{s}\parallel }_0\ge 2k\mathrm{or}{\parallel \mathbf{s}\parallel }_0=0\hfill \end{array}\right..\hfill \end{array}$$

To compare two solutions, the Pareto dominance relationship and the Pareto optimality are defined as follows:

Definition 1

(Pareto dominance). Let ${\mathbf{s}}_1$ and ${\mathbf{s}}_2$ be two solutions, then

• (Weak dominance) ${\mathbf{s}}_1$ weakly dominates ${\mathbf{s}}_2$ (${\mathbf{s}}_1⪯{\mathbf{s}}_2$), if and only if $f\left({\mathbf{s}}_1\right)\le f\left({\mathbf{s}}_2\right)$ and $c\left({\mathbf{s}}_1\right)\le c\left({\mathbf{s}}_2\right)$;
• (Dominance) ${\mathbf{s}}_1$ dominates ${\mathbf{s}}_2$ (${\mathbf{s}}_1\prec {\mathbf{s}}_2$), if and only if ${\mathbf{s}}_1⪯{\mathbf{s}}_2$ and $\left(f\left({\mathbf{s}}_1\right)<f\left({\mathbf{s}}_2\right)\right.$ or $\left.c\left({\mathbf{s}}_1\right)<c\left({\mathbf{s}}_2\right)\right)$.

Definition 2

(Pareto optimality). We say ${\mathbf{s}}^{★}$ is a Pareto optimal solution with respect to $\mathcal{M}$, if there is no solution $\mathbf{s}\in \mathcal{M}$ such that $\mathbf{s}\prec {\mathbf{s}}^{★}$.

To summarize, a solution ${\mathbf{s}}_1$ is said to weakly dominates ${\mathbf{s}}_2$ if ${\mathbf{s}}_1$ performs not worse than ${\mathbf{s}}_2$ for all metrics, and for two solutions such that ${\mathbf{s}}_1⪯{\mathbf{s}}_2$, we say ${\mathbf{s}}_1$ dominates ${\mathbf{s}}_2$ if there is at least one metric for which ${\mathbf{s}}_1$ strictlly outperforms ${\mathbf{s}}_2$. ${\mathbf{s}}^{★}$ is said to be a Pareto optimal solution with respect to $\mathcal{M}$ if there is no solution in $\mathcal{M}$ that dominates ${\mathbf{s}}^{★}$.

3.1. Pareto Optimization for Subset Selection (Poss)

POSS is an evolutionary algorithm for subset selection, and its overall procedure is shown in Algorithm 1. Denote $\mathcal{T}=2^{\mathcal{V}}$ as the set of all solutions, $\mathcal{M}\subseteq \mathcal{T}$ as the set of all solutions found so far. POSS maintains a set $\mathcal{P}\subseteq \mathcal{M}$ of all Pareto optimal solutions with respect to $\mathcal{M}$. At each step, POSS first randomly selects a solution from the set of Pareto optimal solutions $\mathcal{P}$, and then adopts the mutation operator to search for a potential Pareto optimal solution. If there is no solution that dominates the generated solution ${\mathbf{s}}^{\prime }$, then POSS includes the generated solution to $\mathcal{P}$, and remove the solutions which are weakly dominated by ${\mathbf{s}}^{\prime }$ from $\mathcal{P}$; Otherwise POSS discards the solution $\mathbf{s}$. Such update strategy in POSS guarantees that if $\mathcal{P}$ is the Pareto optimal solution set at current iteration, then the updating operation does not change the optimality of $\mathcal{P}$. After T iterations, POSS outputs a solution ${\mathbf{s}}^{★}\in \mathcal{P}$ with the minimal reconstructed error and a cardinality smaller than k, i.e., ${\mathbf{s}}^{★}=arg{min}_{\mathbf{s}\in \mathcal{P},{\parallel \mathbf{s}\parallel }_0\le k}f\left(\mathbf{s}\right)$.

Algorithm 1 POSS

Input:  An universal set $\mathcal{V}$, $f:2^{\mathcal{V}}\to \mathbb{R}$, $k\in \left[n\right]$, $T\in {\mathbb{N}}_{+}$.

Output:  a subset $\mathcal{S}\subseteq \mathcal{V}$ with at most k nodes.

1:  $\mathbf{s}={\left\{0\right\}}^n$, $\mathcal{P}=\left\{\mathbf{s}\right\}$.

2:  for $t=1,2,\cdots ,T$do

3:    Select $\mathbf{s}$ from $\mathcal{P}$ uniformly at random.

4:    Generate ${\mathbf{s}}^{\prime }$ by bit-wise flipping with probability $\frac{1}{n}$.

5:    if $\nexists \mathbf{z}\in \mathcal{P}$ such that $\mathbf{z}\prec {\mathbf{s}}^{\prime }$ then

6:      $\mathcal{P}=(\mathcal{P}\backslash \{\mathbf{z}\in \mathcal{P}|{\mathbf{s}}^{\prime }⪯\mathbf{z}\})\cup \left\{{\mathbf{s}}^{\prime }\right\}$.

7:    end if

8:  end for

9:  ${\mathbf{s}}^{★}={argmin}_{\mathbf{s}\in \mathcal{P},{\parallel \mathbf{s}\parallel }_0\le k}f\left(\mathbf{s}\right)$.

3.2. Proposed Pareto Optimiazation for Graph Signal Sampling (Pogss)

However, due to the general $\mathcal{O}\left(n^3\right)$ complexity of the matrix inversion operation, the evaluation step of $f\left(\mathbf{s}\right)$ is very expensive in general, which means we cannot efficiently compare two solutions. Please note that at each iteration the new solution is generated from a solution evaluated before, we can accelerate the evaluation of the objective function by leveraging the information of the previous solution. More explicitly, by using the matrix inversion lemma [20] we deduced the following lemma:

Lemma 1.

Let $\mathcal{S}\subseteq \mathcal{V}$, then for all $v_1\in \mathcal{V}\backslash \mathcal{S},v_2\in \mathcal{S}$, we have

$$\begin{array}{cc}\hfill & {\overline{\mathbf{K}}}_{\mathcal{S}\cup \left\{v_1\right\}}={\overline{\mathbf{K}}}_{\mathcal{S}}-\frac{{\overline{\mathbf{K}}}_{\mathcal{S}}{\mathbf{v}}_{v_1}{\mathbf{v}}_{v_1}^T{\overline{\mathbf{K}}}_{\mathcal{S}}}{{\sigma }^2+{\mathbf{v}}_{v_1}^T{\overline{\mathbf{K}}}_{\mathcal{S}}{\mathbf{v}}_{v_1}}\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill & {\overline{\mathbf{K}}}_{\mathcal{S}\backslash \left\{v_2\right\}}={\overline{\mathbf{K}}}_{\mathcal{S}}+\frac{{\overline{\mathbf{K}}}_{\mathcal{S}}{\mathbf{v}}_{v_2}{\mathbf{v}}_{v_2}^T{\overline{\mathbf{K}}}_{\mathcal{S}}}{{\sigma }^2-{\mathbf{v}}_{v_2}^T{\overline{\mathbf{K}}}_{\mathcal{S}}{\mathbf{v}}_{v_2}}\hfill \end{array}$$

(10)

Let ${\mathbf{t}}_I={\mathbf{s}}^{\prime }\backslash \mathbf{s}=\{t_{I_1},\dots ,t_{I_l}\}$ be the elements to be included at the current iteration, ${\mathbf{t}}_E=\mathbf{s}\backslash {\mathbf{s}}^{\prime }=\{t_{E_1},\dots ,t_{E_m}\}$ denote the elements to be excluded, and ${\mathbf{t}}_R=\mathbf{s}\cap {\mathbf{s}}^{\prime }$ denote the elements that will be retained. Instead of directly computing the ${\overline{\mathbf{K}}}_{{\mathbf{s}}^{\prime }}$, we propose to compute ${\overline{\mathbf{K}}}_{\mathbf{s}\cup {\mathbf{t}}_I}$ firstly, and then compute the desired ${\overline{\mathbf{K}}}_{{\mathbf{s}}^{\prime }}={\overline{\mathbf{K}}}_{(\mathbf{s}\cup {\mathbf{t}}_I)\backslash {\mathbf{t}}_E}$. By using Lemma 1, we can easily compute ${\overline{\mathbf{K}}}_{\mathbf{s}\cup {\mathbf{t}}_I}$ by including the elements in ${\mathbf{t}}_I$ to $\mathbf{s}$ one by one, and then compute ${\overline{\mathbf{K}}}_{{\mathbf{s}}^{\prime }}$ by excluding the elements in ${\mathbf{t}}_E$ from $\mathbf{s}\cup {\mathbf{t}}_I$ separately. The full procedure of POGSS is described in Algorithm 2, and the solution flipping algorithm for acceleration is presented in Algorithm 3.

Denote $N_t=|{\mathbf{t}}_I|+|{\mathbf{t}}_E|$ as the number of bits that will be flipped at iteration t, then we have $\mathbb{E}\left(N_t\right)=1$, which reveals the fact that we are expected to flip only 1 bit at each iteration. Please note that the evaluation of (9) and (10) can be done at a cost of $\mathcal{O}\left(n^2\right)$, so the cost of evaluating the objective function in Algorithm 2 is expected to be $\mathcal{O}\left(n^2\right)$, while the naive matrix inversion update requires a cost of $\mathcal{O}\left(n^3\right)$. Therefore, this proposed procedure significantly accelerates the algorithm.

Algorithm 2 POGSS

Input:  $\mathsf{\Sigma }$, $\mathbf{V}$, integer $k\in \left[n\right]$, $T\in {\mathbb{N}}_{+}$.

Output:  a subset ${\mathcal{S}}^{★}\subseteq \mathcal{V}$ with at most k nodes.

1:  $\mathbf{s}={\left\{0\right\}}^n$, ${\overline{\mathbf{K}}}_{\mathbf{s}}=\mathsf{\Sigma }$, $\mathcal{P}=\left\{\mathbf{s}\right\}$.

2:  for $t=1,2,\cdots ,T$do

3:    Select $\mathbf{s}$ from $\mathcal{P}$ uniformly at random.

4:    Generate ${\mathbf{s}}^{\prime }$ by bit-wise flipping with probability $\frac{1}{n}$.

5:    $\mathrm{Compute}{\overline{\mathbf{K}}}_{{\mathbf{s}}^{\prime }}\mathrm{by}\mathrm{invoking}\mathrm{Algorithm},\mathrm{then}\mathrm{evaluate}\mathrm{the}\mathrm{objective}\mathrm{function}f\left({\mathbf{s}}^{\prime }\right)=Tr\left(\mathbf{M}{\overline{\mathbf{K}}}_{{\mathbf{s}}^{\prime }}\right).$

6:    if $\nexists \mathbf{z}\in \mathcal{P}$ such that $\mathbf{z}\prec {\mathbf{s}}^{\prime }$ then

7:      $\mathcal{P}=(\mathcal{P}\backslash \{\mathbf{z}\in \mathcal{P}|{\mathbf{s}}^{\prime }⪯\mathbf{z}\})\cup \left\{{\mathbf{s}}^{\prime }\right\}$.

8:    end if

9:  end for

10:  Output ${\mathbf{s}}^{★}={argmin}_{\mathbf{s}\in \mathcal{P},{\parallel \mathbf{s}\parallel }_0\le k}f\left(\mathbf{s}\right)$.

Algorithm 3 Solution Flipping

Input:  $\mathbf{s},{\overline{\mathbf{K}}}_{\mathbf{s}},{\mathbf{s}}^{\prime }$.

Output:  ${\overline{\mathbf{K}}}_{{\mathbf{s}}^{\prime }}$.

1:  Initialize ${\mathbf{s}}_0=\mathbf{s}$, ${\overline{\mathbf{K}}}_{{\mathbf{s}}_0}={\overline{\mathbf{K}}}_{\mathbf{s}}$.

2:  Compute the difference set ${\mathbf{t}}_I={\mathbf{s}}^{\prime }\backslash =\{t_{I_1},\cdots ,t_{I_l}\}$ and ${\mathbf{t}}_E=\mathbf{s}\backslash {\mathbf{s}}^{\prime }=\{t_{E_1},\cdots ,t_{E_m}\}$.

3:  for $i=1\cdots ,l$do

4:    ${\mathbf{s}}_i={\mathbf{s}}_{i-1}\cup \left\{t_{I_i}\right\}$.

5:    Compute ${\overline{\mathbf{K}}}_{{\mathbf{s}}_i}={\overline{\mathbf{K}}}_{{\mathbf{s}}_{i-1}\cup \left\{t_{I_i}\right\}}$ by using (9).

6:  end for

7:  for $i=1\cdots ,m$do

8:    ${\mathbf{s}}_{l+i}={\mathbf{s}}_{l+i-1}\backslash \left\{t_{E_i}\right\}$.

9:    Compute ${\overline{\mathbf{K}}}_{{\mathbf{s}}_{l+i}}={\overline{\mathbf{K}}}_{{\mathbf{s}}_{l+i-1}\backslash \left\{t_{E_i}\right\}}$ by using (10).

10:  end for

11:  Output ${\overline{\mathbf{K}}}_{{\mathbf{s}}^{\prime }}={\overline{\mathbf{K}}}_{{\mathbf{s}}_{l+m}}$.

4. Theoretical Analysis

In this section, we theoretically analyze the performance of POGSS, including the Pareto optimality of $\mathcal{P}$ and the overall reconstruction error of POGSS.

We first show in Proposition 1 that the subset $\mathcal{P}$ is the set of all Pareto optimal solutions with respect to $\mathcal{M}$.

Proposition 1

(Pareto optimality of $\mathcal{P}$). At any iteration of the algorithm 2, let $\mathcal{M}$ be the set of all solutions found so far, then

• for all $\mathbf{s}\in \mathcal{P}$, there is no ${\mathbf{s}}^{\prime }\in \mathcal{M}$ such that ${\mathbf{s}}^{\prime }\prec \mathbf{s}$,
• for all $\mathbf{s}\in (\mathcal{M}\backslash \mathcal{P})$, there is a solution ${\mathbf{s}}^{\prime }\in \mathcal{M}$ such that ${\mathbf{s}}^{\prime }\prec \mathbf{s}$.

We then analyze the performance of POGSS by introducing the supermodularity ratio. A set function $f:2^{\mathcal{V}}\to {\mathbb{R}}_{+}$ is said to be a supermodular function, if $f(\mathcal{A}\cup \{u\left\}\right)-f\left(\mathcal{A}\right)\le f(\mathcal{B}\cup \{u\left\}\right)-f\left(\mathcal{B}\right)$ holds for any $\mathcal{A}\subseteq \mathcal{B}\subseteq \mathcal{V}$ and $v\in \mathcal{V}\backslash \mathcal{B}$. For a general criterion f, Definition 3 defines how close it is to a supermodular function, i.e., the supermodularity ratio ${\gamma }_{\mathcal{S},k}\left(f\right)$ [21]. It is easy to prove that f is supermodular if and only if ${\gamma }_{\mathcal{S},k}\left(f\right)\ge 1$ holds for any $\mathcal{S}$ and k.

Definition 3

(Supermodularity ratio). Let f be a non-negative set function. The submodularity ratio of f with respect to a set $\mathcal{S}$ and a parameter $k\ge 1$ is

$$\begin{array}{c}\hfill {\gamma }_{\mathcal{S},k}\left(f\right)=\underset{\mathcal{L}\subseteq \mathcal{S},\mathcal{U}:\left|\mathcal{U}\right|\le k,\mathcal{L}\cap \mathcal{U}=⌀}{min}\frac{f(\mathcal{L}\cup \mathcal{U})-f\left(\mathcal{L}\right)}{{\sum }_{u\in \mathcal{U}}(f(\mathcal{L}\cup \left\{u\right\})-f\left(\mathcal{L}\right))}\end{array}$$

(11)

The most time-consuming operation in POGSS is the evaluation of the objective function after bit flipping. So here comes a question: how many bit flipping operations we need to obtain a desired solution? We provide an approximation upper bound of the optimal reconstruction MSE with $\mathcal{O}\left(k^{2}n\right)$ bits flipped in Theorem 1. Please note that the greedy algorithm guarantees $(1-e^{-{\gamma }_{{\mathbf{s}}^G,k}})·OPT+e^{-{\gamma }_{{\mathbf{s}}^G,k}}f\left(\mathbf{0}\right)$ performance, where ${\mathbf{s}}^G$ is the solution given by the greedy algorithm, POGSS achieves nearly the best known approximation guarantee as greedy algorithms do in quadratic time.

Theorem 1.

Define ${\gamma }_{min}={min}_{{\mathbf{s}:\parallel \mathbf{s}\parallel }_0=k-1}{\gamma }_{\mathbf{s},k}$, let N be the total number of flipped bits, $OPT$ be the optimal value of (7). Then POGSS finds a solution $\mathbf{s}$ such that

$$\begin{array}{c}\hfill f\left(\mathbf{s}\right)\le (1-e^{-{\gamma }_{min}})·OPT+e^{-{\gamma }_{min}}f\left(\mathbf{0}\right),\end{array}$$

with

$$\begin{array}{c}\hfill \mathbb{E}\left(N\right)\le 2k^{2}exp\left(-\frac{n}{n-0.5}\right)n\le 2e{k}^{2}n.\end{array}$$

The proof of Theorem 1 relies on Lemma 2 and 3 stated bellow. The complete proof is similar to the proof of theorem 14.1 in [22], which is omitted due to the limited space.

Lemma 2.

For any $n\ge 2$,

$$\begin{array}{c}\hfill {\left(1-\frac{1}{n}\right)}^{n-1}\ge e^{-(n-0.5)/n}\ge e^{-1}\end{array}$$

holds.

Lemma 3

([16]). $\forall \mathbf{s}\in {\{0,1\}}^n$, there exsits one nodes $v\in \mathbf{s}$ such that

$$\begin{array}{c}\hfill f\left(\mathbf{s}\right)-f(\mathbf{s}\cup \left\{v\right\})\ge \frac{{\gamma }_{\mathbf{s},b}}{b}\left(f\left(\mathbf{s}\right)-OPT\right)\end{array}$$

Please note that the approximation upper bound in Theorem 1 involves ${\gamma }_{min}$, we show in Theorem 2 that ${\gamma }_{min}$ is also bounded.

Proposition 2

(Lower bound for ${\gamma }_{min}$). The supermodularity ratio satisfies

$$\begin{array}{c}\hfill {\gamma }_{min}\ge \frac{1}{k}.\end{array}$$

This conclusion is natural and obvious, since the equality holds when $f\left(\mathcal{L}\right)-f(\mathcal{L}\cup \mathcal{U})={max}_{u\in \mathcal{U}}f\left(\mathcal{L}\right)-f(\mathcal{L}\cup \left\{u\right\})$, and is also relatively loose. However, it is still meaningful because it is independent of the graph structure and the Signal-to-Noise Ratio (SNR), so it is tighter than the bound in [10] in high SNR scenarios.

5. Experiments

In this section, we conduct several experiments to evaluate the performance of POGSS. We first state the experimental settings, and then report the experimental results.

5.1. Experimental Settings

Two types of undirected graphs are adopted for evaluation:

• The Erdos-Renyi (ER) graph: For ER graphs, any two nodes are connected with probability p, i.e., $\mathbb{P}({\mathbf{W}}_{i,j}={\mathbf{W}}_{j,i}=1)=p$, $\mathbb{P}({\mathbf{W}}_{i,j}={\mathbf{W}}_{j,i}=0)=1-p$.
• The Gaussian graph: For Gaussian graphs, we uniformly sample n points ${\left\{{\mathbf{p}}_i\right\}}_{i=1}^n$ in a unit square ${[0,1]}^2$. The edges are first connected with weights

  $$\begin{array}{c}\hfill {\mathbf{W}}_{i,j}={\mathbf{W}}_{j,i}=exp\left(-\frac{{\parallel {\mathbf{p}}_i-{\mathbf{p}}_j\parallel }_2^2}{2s^2}\right),\end{array}$$

  and we then set the weights which are smaller to some threshold $W_T$ to 0.

In our experiments, the number of nodes $\left|\mathcal{V}\right|=n$ is set to 100 for all graphs, and we set $p=0.2$ for ER graphs, set $s=0.3,W_T=0.6$ for Gaussian graphs.

The graph signal $\mathbf{x}$ is sampled from a Gaussian distribution:

$$\begin{array}{c}\hfill \mathbf{x}\sim \mathcal{N}(\mathbf{0},{\mathbf{V}}_{\mathcal{K}}\overline{\mathsf{\Sigma }}{\mathbf{V}}_{\mathcal{K}}^T)\end{array}$$

(12)

where ${\mathbf{V}}_{\mathcal{K}}$ is the matrix of graph Fourier bases, $\overline{\mathsf{\Sigma }}$ is the covariance matrix of ${\overline{\mathbf{x}}}_{\mathcal{K}}$. In our experiments, $\overline{\mathsf{\Sigma }}$ is generated by $\overline{\mathsf{\Sigma }}=\mathbf{P}{\mathbf{P}}^T$, where $\mathbf{P}\in {\mathbb{R}}^{\left|\mathcal{K}\right|\times \left|\mathcal{K}\right|}$ is a random matrix with ${\mathbf{P}}_{i,j}\sim \mathcal{N}(0,1)$. The additive Gaussian noise $\mathbf{n}$ is adopted in our experiments, and the noise power ${\sigma }^2$ is set to $0.01$.

We compare our methods with several algorithms, including the state-of-the-art greedy algorithm, the randomized greedy algorithm, and the convex relaxation algorithm. In all experiments, the number of iterations of POGSS is set to $\lceil 2e{k}^{2}n\rceil$ according to Theorem 1. The naive greedy graph siganl sampling algorithm [10] simply selects a node with the largest marginal gain at each iteartion. Assume subset of nodes obtained by the greedy algorithm at the current iteration is $\mathcal{S}$, then the node $v=arg{max}_{v\in \mathcal{V}\backslash \mathcal{S}}f(\mathcal{S}\cup \left\{v\right\})$ is included to $\mathcal{S}$. The randomized greedy algorithm [11] uses the similar strategy to the greedy algorithm, but accelerates the algorithm by selecting the best node from a random subset. More explicitly, it first samples a subset $\mathcal{R}\subset \mathcal{V}\backslash \mathcal{S}$ with size $-n/klog\left(ϵ\right)$ uniformly, where $0<ϵ<1$ is a hyper-parameter, then includes the node $v=arg{max}_{v\in \mathcal{R}}f(\mathcal{S}\cup \left\{v\right\})$ to $\mathcal{S}$. Recall the original problem of graph signal sampling (7)

$$\begin{array}{c}\hfill {\mathbf{s}}^{★}=arg\underset{\mathbf{s}\in {\{0,1\}}^n}{min}f\left(\mathbf{s}\right)=Tr\left(\mathbf{M}\left({\overline{\mathsf{\Sigma }}}^{-1}+{\sigma }^{-2}\sum _{i=0}^n{s}_i{\mathbf{v}}_i{\mathbf{v}}_i^T\right)\right),\mathrm{s}.\mathrm{t}.{\parallel \mathbf{s}\parallel }_1\le k,\end{array}$$

(13)

since the original problem is neither differentiable nor convex, we relax this problem to a convex optimization problem [23]

$$\begin{array}{c}\hfill {\mathbf{d}}^{★}=arg\underset{\mathbf{d}\in {[0,1]}^n}{min}f\left(\mathbf{d}\right),\mathrm{s}.\mathrm{t}.{\parallel \mathbf{d}\parallel }_1\le k,\end{array}$$

(14)

then let the largest k entries in ${\mathbf{d}}^{★}$ to be included to the sampling set.

5.2. Experimental Results

We first evaluate the algorithms under the $\mathcal{K}$-bandlimited signal assumption. Under this assumption, the entries in $\overline{\mathbf{x}}$ indexed by $\mathcal{K}$ are zero, thus $\mathbf{x}$ can be perfectly represented by only $\left|\mathcal{K}\right|$ graph Fourier bases. In the following experiments we set $\left|\mathcal{K}\right|=40$ and use the first 40 graph Fourier bases to form ${\mathbf{V}}_{\mathcal{K}}$.

We set the subset size k from 10 to 100 with stepsize 10, then independently run the simulation 10 times and report the average optimal reconstruction MSE of the obtained subset of each algorithm. Figure 2 and Figure 3 illustrate the results obtained on ER graphs and Gaussian graphs respectively. From Figure 2a and Figure 3a we observe that POGSS and greedy-based algorithms achieve near-optimal performance, the performance convex relaxation method is worse than other method, and POGSS outperforms others for any subset size. We note that when the subset size k is approaching $\left|\mathcal{K}\right|=40$ the MSE decreases dramatically, and when $k>\left|\mathcal{K}\right|=40$ (shown in the right-hand side of the vertical dotted line) all of the methods provide the solutions with very small reconstruction MSE. Figure 2b and Figure 3b plot the convergence curve of POGSS when $k=20$. We can see that the MSE decreases very fast at first, and then becomes flat when $k>500$. However, it keeps decreasing throughout the whole process. To be precise, POGSS reaches its’ final solution at the 2144-th iteration, while $T=\lceil 2e{k}^{2}n\rceil =2174$.

We next evaluate the algorithms for non-bandlimited graph signals, i.e., ${\mathbf{V}}_{\mathcal{K}}=\mathbf{V}$. Similar to the bandlimited scenario, we set the subset size from 10 to 100 with step size 10 and present the average optimal reconstruction MSE of the obtained subset of each algorithm. Figure 4 and Figure 5 show the results obtained on ER graphs and Gaussian graphs respectively. From Figure 4a and Figure 5a we observe the similar results to the ones in Figure 2a and Figure 3a, where POGSS still achieves the best performance. It is worth noting that there is no “turning point” in Figure 4 and Figure 5, because the signals are non-bandlimited, and cannot be accurately reconstructed from partial observations.

In Figure 6 we compare the running time of different sampling set sizes. We observe that POGSS takes the most time to converge, which is as expected since Theorem 1 reveals that POGSS takse $2e{k}^{2}n$ iterations to converge, while greedy algorithms require only k iterations to do that (the time consumption of each method to finish one iteration is almost the same). Although POGSS takes the most time to find a desired solution, it is still practicable for two reasons:

• if we fit the running time of POGSS with a quadratic function, then we have $T\approx 0.26·k^2\left(\sec \right)$, thus POGSS only takes only $\Delta T\approx 5.15·k+25.77\left(\sec \right)$ when we add 10 nodes to the current sampling set, we think it is worthwhile to find a better solution with such a time consumption.
• In real world applications of graph signal sampling, e.g., [8], the sampling procedure is offline and the reconstruction procedure is online. We only need to sample the graph once then the entire graph signal can be always reconstructed with a small error, while the reconstruction time depends only on the size of the sampling set, rather than the sampling method.

We also provide a visual demonstration for our method. We use the stanford bunny data [6] to obtain the graph. The nodes are scanned from a 3D model, and the edges are obtained by connecting each node with its’ four nearest neighbors. The smooth graph signal is first sampled from a Gaussian distribution, and then filtered by a heat diffusion low-pass filter [24]. The resulting graph Fourier coefficients after filtering are

$$\begin{array}{c}\hfill {\widehat{\lambda }}_i=h\left({\lambda }_i\right)=exp(-t{\lambda }_i),\end{array}$$

where t is the scaling parameter and is set to 10 in our experiment. As displayed in Figure 7a is the original smooth graph signal with Gaussian noise, the noise power is ${\sigma }^2=0.01$. There are $n=453$ nodes in the original graph, and we sample a subset with $k=300$ nodes by using POGSS to obtain Figure 7b, note that all of the edges are from the original graph. Figure 7c illustrates the optimally recovered graph signal by using (3), and the corresponding reconstruction error is ${\mathrm{MSE}}_{\mathcal{S}}^{★}=137.2$.

6. Conclusions

In this paper, we consider the graph signal sampling problem, which can be formulated as a subset selection problem. The evolutionary Pareto optimization for graph signal sampling (POGSS) is proposed for solving this problem. An efficient procedure is also proposed to accelerate the evaluation of the objective function. Theoretical analysis shows that POGSS, in polynomial time, achieves the best approximation upper bound of $(1-e^{-{\gamma }_{min}})·OPT+e^{-{\gamma }_{min}}f\left(\mathbf{0}\right)$ as the state-of-the-art greedy algorithm does, and an lower bound of ${\gamma }_{min}$ is built as well. Simulation results demonstrate that POGSS outperforms the previous greedy algorithms.