Efficient Algorithms for the Maximum Sum Problems

Abstract

We present efficient sequential and parallel algorithms for the maximum sum (MS) problem, which is to maximize the sum of some shape in the data array. We deal with two MS problems; the maximum subarray (MSA) problem and the maximum convex sum (MCS) problem. In the MSA problem, we find a rectangular part within the given data array that maximizes the sum in it. The MCS problem is to find a convex shape rather than a rectangular shape that maximizes the sum. Thus, MCS is a generalization of MSA. For the MSA problem, $O\left(n\right)$ time parallel algorithms are already known on an $(n,n)$ 2D array of processors. We improve the communication steps from $2n-1$ to n, which is optimal. For the MCS problem, we achieve the asymptotic time bound of $O\left(n\right)$ on an $(n,n)$ 2D array of processors. We provide rigorous proofs for the correctness of our parallel algorithm based on Hoare logic and also provide some experimental results of our algorithm that are gathered from the Blue Gene/P super computer. Furthermore, we briefly describe how to compute the actual shape of the maximum convex sum.

1. Introduction

We face a challenge to process a big amount of data in the age of information explosion and big data [1]. As the end of Moore’s law comes in sight, however, the extra computing power is unlikely obtained from a single processor computer [2]. Parallel computing is obviously a direction to faster computation. However, efficient utilization of the parallel architecture is not an effortless task. New algorithms often need to be designed for specific problems on specific hardware.

In this paper, we investigate the maximum subarray (MSA) problem and the maximum convex sum (MCS) problem, the latter being a generalization of the former. We design efficient parallel algorithms for both problems and an improved sequential algorithm for the MCS problem.

The MSA problem is to find a rectangular subarray in the given two-dimensional (2D) data array that maximizes the sum in it. This problem has wide applications from image processing to data mining. An example application from image processing is to find the most distinct spot, such as brightest or darkest, in the given image. If all pixel values are non-negative, solving the MSA problem will return the trivial solution of the whole array. Thus, we normalize the input image (e.g., by subtracting the mean value), such that the relatively brighter spots will have positive sums, while the relatively darker spots will have negative sums. Then, solving the MSA problem on the normalized image can give us the brightest spot in the image. In data mining, suppose we spread the sales amounts of some product on a two-dimensional array classified by customer ages and annual income. Then, after normalizing the data, the maximum subarray corresponds to the most promising customer range.

We now consider the MCS problem that maximizes the sum in a convex shape. The definition of the word “convex” is not exactly the same as that in geometry. We define the convex shape as the joining of a W-shape and an N-shape, where W stands for widening and N for narrowing. We call this shape the $WN$-convex shape or $WN$-shape for short. In W, the top boundary goes up or stays horizontal when scanned from left to right, and the bottom boundary goes down or stays horizontal. Fukuda et al. defines a more general rectilinear shape [3], but for simplicity, we only consider the $WN$-shape. The paper is mainly devoted to computing the maximum sum, and one section is devoted to the computation of the explicit convex shape that provides the sum.

We give an example to illustrate how solving the MCS problem can provide a much more accurate data analysis compared to solving the MSA problem. We compared the size of the hole in the ozone layer over Antarctica between 2006 and 2011 by solving both the MSA problem and the MCS problem on the normalized input images (Figure 1a,d, respectively) of the Antarctic region. Figure 1b,e is from solving the MSA problem on the input images, while Figure 1c,f is from solving the MCS problem on the same set of input images. Solving the MCS problem clearly provides a much more accurate representation of the ozone hole. The numeric value returned by the MCS is also 22%~24% greater than that by the MSA on both occasions. Intuitively similar gains from the $WN$-shape compared to the rectangular shape can be foreseen for other types of data analysis.

In this paper, we assume that the input data array is an $(n,n)$ square two-dimensional (2D) array. It is a straightforward exercise to extend the contents of this paper to $(n,m)$ rectangular 2D input data arrays.

The typical algorithm for the MSA problem by Bentley [4] takes $O\left(n^3\right)$ time on a sequential computer. This has been improved to a slightly sub-cubic time bound by Tamaki and Tokuyama [5] and also Takaoka [6]. For the MCS problem, an algorithm with $O\left(n^3\right)$ time is given by Fukuda et al. [3], and an algorithm with a sub-cubic time bound is not known.

Takaoka discussed a parallel implementation to solve the MSA problem on a PRAM [6]. Bae and Takaoka implemented a range of parallel algorithms for the MSA problem on an $(n,n)$ 2D mesh array architecture based on the row-wise or column-wise prefix sum [7,8,9]. A parallel algorithm for the MSA problem was also implemented on the BSP/CGMarchitecture, which has more local memory and communication capabilities with remote processors [10].

In this paper, we implement algorithms for the MSA and MCS problems based on the column-wise prefix sum on the 2D mesh architecture, as shown by Figure 2. This architecture is also known as a systolic array, where each processing unit has a constant number of registers and is permitted to only communicate with directly-connected neighbours. This seemingly inflexible architecture is well suited to be implemented on ASICs or FPGAs.

Our most efficient parallel algorithms complete the computation in $n-1$ communication steps for the MSA problem and $4n$ communication steps for the MCS problem, respectively. In each step, all cells execute a constant number of statements in parallel. Thus, our algorithms are cost optimal with respect to the cubic time sequential algorithms [3,4].

We give a formal proof for the parallel algorithm for computing the maximum W (a part of computing the MCS problem). The proof is based on the space-time invariants defined on the architecture.

In a 2D array architecture, some processors to the right are sitting idle in the early stage of computation waiting for inputs to arrive. This is because the data only flows from left to right and from up to down (Figure 2). If appropriate, we attempt to maximize the throughput by extending data flows to operate in four directions. This technique reduces the number of communication steps by a constant factor.

Algorithms are given by pseudocode.

2. Parallel Algorithms for the MSA Problem

In this section, we improve parallel algorithms for the MSA problem on a mesh array and achieve the optimal n communication steps. Furthermore, we show how the programming techniques in this section lead to efficient parallel algorithms for the MCS problem in the later sections.

2.1. Sequential Algorithm

The computation in Algorithm 1 [8] proceeds with the strip of the array from position k to position i. See Figure 3. The variable $column\left[i\right]\left[j\right]$ is the sum of array elements in the j-th column from position k to position i in array a. The variable $sum\left[i\right]\left[j\right]$, called a prefix-sum, is the sum of the strip from Position 1 to position j. Within this strip, variable j sweeps to compute $column\left[i\right]\left[j\right]$ by adding $a\left[i\right]\left[j\right]$ to $column[i-1]\left[j\right]$. Then, the prefix sum of this strip from Position 1 to position j is computed by adding $column\left[i\right]\left[j\right]$ to $sum\left[i\right][j-1]$. The variable $min\_sum\left[i\right]\left[j\right]$ is the minimum prefix sum of this strip from Position 1 to position j. If the current $sum$ is smaller than $min\_sum\left[i\right]\left[j\right]$, $min\_sum\left[i\right]\left[j\right]$ is replaced by it. $sol\left[i\right]\left[j\right]$ is the maximum sum in this strip so far found from Position 1 to position j. It is computed by taking the maximum of $sol\left[i\right][j-1]$ and $sum\left[i\right]\left[j\right]-min\_sum\left[i\right]\left[j\right]$, expressed by $max\_sum$ in the figure. After the computation for this strip is over, the global solution, S, is updated by $sol\left[i\right]\left[n\right]$. This computation is done for all possible i and k, taking $O\left(n^3\right)$ time.

Algorithm 1 MSA: sequential.

1: for $1\le i\le n$ do 2:     $min\_sum\left[i\right]\left[0\right]\leftarrow \infty ;sum\left[i\right]\left[0\right]\leftarrow 0;sol\left[i\right]\left[0\right]\leftarrow -\infty ;$ 3: end for 4: $S\leftarrow -\infty$; 5: for $k\leftarrow 1$ to n do 6:     for $j\leftarrow 1$ to n do 7:         $column[k-1]\left[j\right]\leftarrow 0$; 8:     end for 9:     for i←k to n do 10:         for $j\leftarrow 1$ to n do 11:            $column\left[i\right]\left[j\right]\leftarrow column[i-1]\left[j\right]+a\left[i\right]\left[j\right];$ 12:            $sum\left[i\right]\left[j\right]\leftarrow sum\left[i\right][j-1]+column\left[i\right]\left[j\right];$ 13:            $min\_sum\left[i\right]\left[j\right]\leftarrow \mathbb{MIN}\left\{sum\right[i\left]\right[j],min\_sum[i\left]\right[j-1]$ }; 14:            $max\_sum\left[i\right]\left[j\right]\leftarrow sum\left[i\right]\left[j\right]-min\_sum\left[i\right]\left[j\right]$; 15:            $sol\left[i\right]\left[j\right]\leftarrow \mathbb{MAX}\{max\_sum[i\left]\right[j],sol[i\left]\right[j-1\left]\right\}$; 16:         end for 17:         if $sol\left[i\right]\left[n\right]>S$ then 18:            $S\leftarrow sol\left[i\right]\left[n\right];$ 19:         end if 20:     end for 21: end for

2.2. Parallel Algorithm 1

Algorithm 2 is a parallel adaptation of Algorithm 1. The following program is executed by a processing unit at the $(i,j)$ grid point, which we refer to as $cell(i,j)$. Each $cell(i,j)$ is aware of its position $(i,j)$. Data flow is from left to right and from top to bottom. The control signals are fired at the left border and propagate right. When the signal arrives at $cell(i,j)$, it accumulates the column sum “$column$”, the sum “$sum$” and updates the minimum prefix sum “$min$”, etc. Figure 4 illustrates the information available to $cell(i,j)$ at step k. At step $2n-1$, $cell(n,n)$ will have computed the maximum sum. We assume that all corresponding instructions in all cells are executed at the same time, that is they are synchronized. We will later make some comments on asynchronous computation.

Algorithm 2 MSA Parallel 1.

Initialization 1: for all $0\le i,j\le n$ in parallel do 2:     $column\left[i\right]\left[j\right]\leftarrow 0;$ $min\left[i\right]\left[j\right]\leftarrow \infty$; 3:     $control\left[i\right]\left[\right[j]\leftarrow 0$; $sum\left[i\right]\left[j\right]\leftarrow 0;$ 4: end for 5: for $0\le i\le n$ in parallel do 6:     $control\left[i\right]\left[0\right]\leftarrow 1;sol\left[i\right]\left[0\right]\leftarrow -\infty ;$ 7: end for   Main 8: for $k\leftarrow 1$ to $2n-1$ do 9:     for all $1\le i,j\le n$ in parallel do 10:         if $control\left[i\right][j-1]=1$ then 11:            $column\left[i\right]\left[j\right]\leftarrow column[i-1]\left[j\right]+a\left[i\right]\left[j\right];$ 12:            $sum\left[i\right]\left[j\right]\leftarrow sum\left[i\right][j-1]+column\left[i\right]\left[j\right];$ 13:            $min\left[i\right]\left[j\right]\leftarrow \mathbb{MIN}\left\{min\right[i\left]\right[j-1],sum[i\left]\right[j\left]\right\};$ 14:            $max\left[i\right]\left[j\right]\leftarrow sum\left[i\right]\left[j\right]-min\left[i\right]\left[j\right]$; 15:            $sol\left[i\right]\left[j\right]\leftarrow \mathbb{MAX}\left\{sol\right[i-1\left]\right[j],sol[i\left]\right[j-1],sol[i\left]\right[j],max[i\left]\right[j\left]\right\};$ 16:            $control\left[i\right]\left[j\right]\leftarrow 1;$ 17:         end if 18:     end for 19: end for

2.3. Parallel Algorithm 2

This algorithm (Algorithm 3) does communication bi-directionally in a horizontal way. For simplicity, we assume n is even. The $(n,n)$ mesh is divided into two halves, left and right. The left half operates in the same way as Algorithm 2. The right half operates in a mirror image, that is control signals go from right to left initiated at the right border. All other data also flow from right to left. At the centre, that is at $(i,n/2)$, $cell(i,n/2)$ performs “$centre\left[i\right]\leftarrow max\left[i\right][n/2]+max\left[i\right][n/2+1]$”, which adds the two values that are the sums of strip regions in the left and right whose heights are equal and, thus, can be added to form a possible solution crossing over the centre. At the end of the k-th iteration, all properties in Algorithm 2 hold on the left half, and the properties in the mirror image hold on the right half. In addition, we have that $centre\left[i\right]$ is the value of the maximum subarray that lies above or that is touching the i-th row and crosses over the centre line.

Algorithm 3 MSA Parallel 2.

Initialization 1: for all $0\le i,j\le n$ in parallel do 2:     $column\left[i\right]\left[j\right]\leftarrow 0;min\left[i\right]\left[j\right]\leftarrow \infty ;$ 3:     $control\left[i\right]\left[\right[j]\leftarrow 0;sum[i\left]\right[j]\leftarrow 0;$ 4: end for 5: for $0\le i\le n$ in parallel do 6:     $control\left[i\right]\left[0\right]\leftarrow 1;control\left[i\right][n+1]\leftarrow 1;$ 7: end for Main 8: for $k\leftarrow 1$ to $(3/2)n-1$ do 9:     for all $1\le i,j\le n$ in parallel do 10:         if $1\le j\le n/2$ then $▹$ left half 11:            if $control\left[i\right][j-1]=1$ then 12:                $column\left[i\right]\left[j\right]\leftarrow column[i-1]\left[j\right]+a\left[i\right]\left[j\right];$ 13:                $sum\left[i\right]\left[j\right]\leftarrow sum\left[i\right][j-1]+column\left[i\right]\left[j\right];$ 14:                $min\left[i\right]\left[j\right]\leftarrow \mathbb{MIN}\left\{min\right[i\left]\right[j-1],sum[i\left]\right[j\left]\right\};$ 15:                $max\left[i\right]\left[j\right]\leftarrow sum\left[i\right]\left[j\right]-min\left[i\right]\left[j\right]$; 16:                $sol\left[i\right]\left[j\right]\leftarrow \mathbb{MAX}\left\{sol\right[i-1\left]\right[j],sol[i\left]\right[j-1],sol[i\left]\right[j],max[i\left]\right[j\left]\right\};$ 17:                $control\left[i\right]\left[j\right]\leftarrow 1;$ 18:            end if 19:         end if 20:         if $n/2+1\le j\le n$ then $▹$ right half 21:            if $control\left[i\right][j+1]=1$ then 22:                $column\left[i\right]\left[j\right]\leftarrow column[i-1]\left[j\right]+a\left[i\right]\left[j\right];$ 23:                $sum\left[i\right]\left[j\right]\leftarrow sum\left[i\right][j+1]+column\left[i\right]\left[j\right];$ 24:                $min\left[i\right]\left[j\right]\leftarrow \mathbb{MIN}\left\{min\right[i\left]\right[j+1],sum[i\left]\right[j\left]\right\};$ 25:                $max\left[i\right]\left[j\right]\leftarrow sum\left[i\right]\left[j\right]-min\left[i\right]\left[j\right]$; 26:                $sol\left[i\right]\left[j\right]\leftarrow \mathbb{MAX}\left\{sol\right[i-1\left]\right[j],sol[i\left]\right[j+1],sol[i\left]\right[j],max[i\left]\right[j\left]\right\};$ 27:                $control\left[i\right]\left[j\right]\leftarrow 1;$ 28:            end if 29:         end if 30:         if $j=n/2$ then $▹$ $cell(i,n/2)$ processes $centre\left[i\right]$ 31:            $centre\left[i\right]\leftarrow max\left[i\right][n/2]+max\left[i\right][n/2+1]$; 32:            if $centre\left[i\right]<centre[i-1]$ then 33:                $centre\left[i\right]\leftarrow centre[i-1];$ 34:            end if 35:         end if 36:     end for 37: end for Finalization step 38: Let $cell(n,n/2)$ do $solution\leftarrow \mathbb{MAX}\left\{sol\right[n\left]\right[n/2],sol[n\left]\right[n/2+1],centre[n\left]\right\};$

The strip $cell(i,j)$ processes is $a[i+j-k,\dots ,i][1,\dots ,j]$ in the left half, and that in the right half is $a[i+n-j+1-k,\dots ,i][j,\dots ,n]$. Thus, the cells $cell(i,n/2)$ and $cell(i,n/2+1)$ processes the strips of the same height in the left half and the right half. Communication steps are measured by the distance from $cell(1,1)$ to $cell(n,n/2)$ or, equivalently, from $cell(1,n)$ to $cell(n,n/2+1)$, which is $(3/2)n-1$. By adding the finalization step, we have $(3/2)n$ for the total communication steps.

2.4. Parallel Algorithm 3

In Algorithm 4, data flow in four directions. The array is divided into two halves; left and right, as in the previous section. Column sums c and prefix sums s accumulate downwards as before, whereas column sums d and prefix sums t accumulate upwards. See Figure 5.

Algorithm 4 MSA Parallel 3: initialization.

1: for all $0\le i,j\le n+1$ in parallel do 2:     $c\left[i\right]\left[j\right]\leftarrow 0;d\left[i\right]\left[j\right]\leftarrow 0;$ 3:     $min\left[i\right]\left[j\right]\leftarrow \infty ;control\left[i\right]\left[\right[j]\leftarrow 0;$ 4:     $s\left[i\right]\left[j\right]\leftarrow 0;t\left[i\right]\left[j\right]\leftarrow 0;$ 5: end for 6: for $0\le i\le n+1$ in parallel do 7:     $control\left[i\right]\left[0\right]\leftarrow 1;control\left[i\right][n+1]\leftarrow 1;$ 8: end for 9: for $k\leftarrow 1$ to $n-2$ do 10:     for all $1\le i,j\le n$ in parallel do 11:         if $1\le j\le n/2$ then 12:            if $control\left[i\right][j-1]=1$ then 13:                $c\left[i\right]\left[j\right]\leftarrow c[i-1]\left[j\right]+a\left[i\right]\left[j\right];$ 14:                $s\left[i\right]\left[j\right]\leftarrow s\left[i\right][j-1]+c\left[i\right]\left[j\right];$ 15:                $d\left[i\right]\left[j\right]\leftarrow d[i+1]\left[j\right]+a\left[i\right]\left[j\right];$ 16:                $t\left[i\right]\left[j\right]\leftarrow t\left[i\right][j-1]+d\left[i\right]\left[j\right];$ 17:                $u\left[i\right]\left[j\right]\leftarrow s\left[i\right]\left[j\right]+t[i+1]\left[j\right]$; 18:                $min\left[i\right]\left[j\right]\leftarrow \mathbb{MIN}\left\{min\right[i\left]\right[j-1],u[i\left]\right[j\left]\right\};$ 19:                $max\left[i\right]\left[j\right]\leftarrow u\left[i\right]\left[j\right]-min\left[i\right]\left[j\right]$; 20:                $sol\left[i\right]\left[j\right]\leftarrow \mathbb{MAX}\left\{sol\right[i-1\left]\right[j],sol[i+1\left]\right[j],sol[i\left]\right[j-1],sol[i\left]\right[j\left]\right\}$; 21:                $sol\left[i\right]\left[j\right]\leftarrow \mathbb{MAX}\left\{sol\right[i\left]\right[j],max[i\left]\right[j\left]\right\};$ 22:                $control\left[i\right]\left[j\right]\leftarrow 1;$ 23:            end if 24:         end if 25:         if $n/2+1\le j\le n$ then 26:            if $control\left[i\right][j+1]=1$ then 27:                $c\left[i\right]\left[j\right]\leftarrow c[i-1]\left[j\right]+a\left[i\right]\left[j\right];$ 28:                $s\left[i\right]\left[j\right]\leftarrow s\left[i\right][j+1]+c\left[i\right]\left[j\right];$ 29:                $d\left[i\right]\left[j\right]\leftarrow d[i+1]\left[j\right]+a\left[i\right]\left[j\right];$ 30:                $t\left[i\right]\left[j\right]\leftarrow t\left[i\right][j+1]+d\left[i\right]\left[j\right];$ 31:                $u\left[i\right]\left[j\right]\leftarrow s\left[i\right]\left[j\right]+t[i+1]\left[j\right]$; 32:                $min\left[i\right]\left[j\right]\leftarrow \mathbb{MIN}\left\{min\right[i\left]\right[j+1],u[i\left]\right[j\left]\right\};$ 33:                $max\left[i\right]\left[j\right]\leftarrow u\left[i\right]\left[j\right]-min\left[i\right]\left[j\right]$; 34:                $sol\left[i\right]\left[j\right]\leftarrow \mathbb{MAX}\left\{sol\right[i-1\left]\right[j],sol[i+1\left]\right[j],sol[i\left]\right[j+1],sol[i\left]\right[j\left]\right\}$; 35:                $sol\left[i\right]\left[j\right]\leftarrow \mathbb{MAX}\left\{sol\right[i\left]\right[j],max[i\left]\right[j\left]\right\};$ 36:                $control\left[i\right]\left[j\right]\leftarrow 1;$ 37:            end if 38:         end if 39:         if $j=n/2$ then $▹$ $cell(i,n/2)$ performs the following 40:            $centre\left[i\right]\leftarrow max\left[i\right][n/2]+max\left[i\right][n/2+1]$; 41:            if $centre\left[i\right]<centre[i-1]$ then 42:                $centre\left[i\right]\leftarrow centre[i-1];$ 43:            end if 44:            if $centre\left[i\right]<centre[i+1]$ then 45:                $centre\left[i\right]\leftarrow centre[i+1];$ 46:            end if 47:         end if 48:     end for 49: end for 50: if $i=n/2$ and $j=n/2$ then $▹$ $cell(n/2,n/2)$ processes $solution$ 51:     $solution\leftarrow \mathbb{MAX}\left\{sol\right[n/2\left]\right[n/2],sol[n/2\left]\right[n/2+1],centre[n/2\left]\right\};$ 52: end if

The structure of Algorithm 4 reveals that at the end of the k-th iteration, $s\left[i\right]\left[j\right]$ is the sum of $a[i+j-k,\dots ,i][1,\dots ,j]$ and $t[i+1]\left[j\right]$ is the sum of $a[i+1,\dots ,i-j+k+1][1,\dots ,j]$. The height of each subarray is $k-j+1$. Since the widths of those two areas are the same, we can have the prefix sum $u\left[i\right]\left[j\right]=s\left[i\right]\left[j\right]+t[i+1]\left[j\right]$ that covers $a[i+j-k,\dots ,i-j+k+1][1,\dots ,j]$, the height of which is $2(k-j+1)$. That is, spending k steps, we can achieve twice as much height as that in Algorithm 3.

The solution array $sol$ is calculated as before, but the result is sent into three directions; up, down and right in the left half and up, down and left in the right half. We have the property that $sol\left[i\right]\left[j\right]$ is the maximum sum in subarray $a[i+j-k,\dots ,i-j+k+1][1,\dots ,j]$ in the left half. Substituting $i=n/2,j=n/2$, and $k=n-1$ yields the subarray $a[1,\dots ,n][1,\dots ,n/2]$. Similarly, $sol[n/2][n/2+1]$ is the maximum sum in the subarray $a[1,\dots ,n][n/2+1,\dots ,n]$. For simplicity, we deal with the maximum subarray whose height is an even number. For a general case, see the note at the end of this section.

The computation proceeds with $n-2$ steps by k and the last step of comparing the results from $cell(n/2,n/2)$ and $cell(n/2+1,n/2)$, resulting in $n-1$ steps in total.

Note that we described the algorithm for the solution whose height is an even number. This fact comes from the assignment statement “$u\left[i\right]\left[j\right]\leftarrow s\left[i\right]\left[j\right]+t[i+1]\left[j\right]$” where the heights of subarrays whose sums are s and t are equal. To accommodate a height of an odd number, we can use the value of t one step before, whose height is one shorter. To accommodate such odd heights, we need to almost double the size of the program by increasing the number of variables.

3. Review of Sequential Algorithm for the MCS Problem

We start from describing a sequential algorithm for W based on column sums, given by Fukuda et al. [3]. We call the rightmost column of a W-shape the anchor column of W. The array portion of general array b, $b[i,\dots ,k][j,\dots ,l]$, is the rectangular portion whose top left corner is $(i,j)$, and the bottom right corner is $(k,l)$. The column $b[i,\dots ,k][j,\dots ,j]$ is abbreviated as $b[i,\dots ,k]\left[j\right]$. In Algorithm 5, $W\left[i\right]\left[k\right]\left[j\right]$ is a W-shape based on the anchor column of $a[k,\dots ,i]\left[j\right]$. The sum of this anchor column is given by $column\left[i\right]\left[k\right]\left[j\right]$.

The computation proceeds with the strip of the array from row k to row i (Figure 6) based on dynamic programming. Within this strip from row k to row i, variable j sweeps to compute $W\left[i\right]\left[k\right]\left[j\right]$ by adding $column\left[i\right]\left[k\right]\left[j\right]$ to $W\left[i\right]\left[k\right][j-1]$ (Case 1 of Figure 7) or extending a W-shape downward or upward by one (Cases 2 and 3 of Figure 7, respectively). Note that the width of the strip is given by $t+1$. That is, we go through from thinner to thicker strips, so that the results for thinner ones are available when needed for Cases 2 and 3. Obviously, Algorithm 5 takes $O\left(n^3\right)$ time.

Algorithm 5 Sequential algorithm for W.

Initialization 1: for all $0\le i,j,k\le n$ do 2:     $W\left[i\right]\left[k\right]\left[j\right]\leftarrow 0$     ▹$W\left[i\right]\left[k\right]\left[j\right]$ stores maximum W for anchor column $column[k,\cdots ,i]\left[j\right]$ 3: end for   Main 4: for $j\leftarrow 1$ to n do 5:     $sum\left[0\right]\left[j\right]\leftarrow 0$ 6:     for $i\leftarrow 1$ to n do 7:         $sum\left[i\right]\left[j\right]\leftarrow sum[i-1]\left[j\right]+a\left[i\right]\left[j\right]$ 8:     end for 9: end for 10: for $t\leftarrow 1$ to $n-1$ do 11:     for $k\leftarrow 1$ to $n-t$ do 12:         $i\leftarrow k+t$ 13:         for $j\leftarrow 1$ to n do 14:            $column\left[i\right]\left[k\right]\left[j\right]\leftarrow sum\left[i\right]\left[j\right]-sum[k-1]\left[j\right]$          ▹ sum of $a[k,\cdots ,i]\left[j\right]$ 15:            $can{d}_1\leftarrow W\left[i\right]\left[k\right][j-1]+column\left[i\right]\left[k\right]\left[j\right]$ 16:            $can{d}_2\leftarrow W[i-1]\left[k\right]\left[j\right]+a\left[i\right]\left[j\right]$ 17:            $can{d}_3\leftarrow W\left[i\right][k+1]\left[j\right]+a\left[k\right]\left[j\right]$ 18:            $W\left[i\right]\left[k\right]\left[j\right]\leftarrow \mathbb{MAX}\{can{d}_1,can{d}_2,can{d}_3\}$ 19:         end for 20:     end for 21: end for

After all W are computed, we compute the N-shape in array N for all anchor columns by a mirror image of the algorithm for W. Then, the finalization of the $WN$-shape is given by Algorithm 6.

Algorithm 6 Combining W and N.

1: $S\leftarrow -\infty$ 2: for $j\leftarrow 1$ to n do 3:     for $k\leftarrow 1$ to n do 4:         for $i\leftarrow k$ to n do 5:            $S\leftarrow \mathbb{MAX}\{S,W[i\left]\right[k\left]\right[j]+N[i\left]\right[k\left]\right[j]-column[i\left]\right[k\left]\right[j\left]\right\}$ 6:         end for 7:     end for 8: end for

Note that the W-shape and the N-shape share the same anchor column, meaning we need to subtract one $column\left[i\right]\left[k\right]\left[j\right]$. This computation is done for all possible i, j and k, taking $O\left(n^3\right)$ time, resulting in $O\left(n^3\right)$ time for the maximum convex sum.

4. Improved Sequential Algorithm

We can observe that Algorithm 5 not only takes $O\left(n^3\right)$ time, but also requires $O\left(n^3\right)$ memory. The reason for this memory requirement is that the algorithm stores maximum W and maximum N for all possible anchor columns.

We can improve the memory requirement of Algorithm 5 significantly to $O\left(n^2\right)$ with a simple modification. Firstly, we observe that there is no reason to keep all maximum W and maximum N for all $O\left(n^3\right)$ possible anchor columns. Instead, we can iterate over the possible anchor column sizes and compute the $O\left(n^2\right)$ maximum W and N for the given anchor column size in each iteration, thereby computing the maximum $WN$ for the given anchor column size in each iteration. Thus, in each iteration, we only need $O\left(n^2\right)$ memory, and there is no need to store maximum W and N values from previous iterations. Note that W and N on shorter columns are available for the t-th iteration.

Algorithm 7 is the pseudocode for achieving the $O\left(n^2\right)$ memory bound. The pseudocode has been simplified, since much of the details have already been provided in Section 3.

Algorithm 7 Sequential algorithm for $MCS$.

1: for $t\leftarrow 1$ to n do 2:     Compute maximum W for all anchor columns of size t 3:     Compute maximum N for all anchor columns of size t 4:     Combine W and N for all anchor columns of size t 5:     Store the current maximum $WN$ 6: end for

We note that the reduction in the memory bound is very significant in practical terms. The difference between $O\left(n^2\right)$ memory and $O\left(n^3\right)$ memory, if we take image processing as an example, is the difference between being able to process a mega-pixel image entirely in memory and having to resort to paging the results in an incomparably slow execution time.

5. Parallel Algorithm for MCS

We now give a parallel algorithm that corresponds to the sequential algorithm described in Section 3. Algorithm 8 is executed by the cell at the $(i,j)$ grid point. Each $cell(i,j)$ is aware of its position $(i,j)$. Data flow is from left to right and from top to down. The control signals are fired at the left border and propagate right. When the signal arrives at $cell(i,j)$, it starts to accumulate the column sum $column$ and update $top$, W and $sol$. The value of $W\left[i\right]\left[j\right]$ at time k is to hold the maximum W-value based on the anchor column in the j-th column from position $i+j-k$ to position i. This is illustrated in Figure 8. The value of $sol\left[i\right]\left[j\right]$ is to hold the best W-value obtained at $cell(i,j)$ so far. The role of $top$ is to bring down the top value of anchor column to $cell(i,j)$ in time. The role of “$old\_W$” is to provide the value of W one step before, with $old\_W\left[i\right]\left[0\right]$ being undefined.

Algorithm 8 Parallel algorithm for W.

Initialization 1: for all $0\le i,j\le n$ in parallel do 2:     $column\left[i\right]\left[j\right]\leftarrow 0$ 3:     $top\left[i\right]\left[j\right]\leftarrow a\left[i\right]\left[j\right]$ 4:     $control\left[i\right]\left[j\right]\leftarrow 0$ 5:     $W\left[i\right]\left[j\right]\leftarrow 0$ 6:     $old\_W\left[i\right]\left[j\right]\leftarrow 0$ 7: end for 8: for $0\le i\le n$ in parallel do 9:     $control\left[i\right]\left[0\right]\leftarrow 0$ 10:     $sol\left[i\right]\left[0\right]\leftarrow -\infty$ 11:     $a\left[i\right]\left[0\right]\leftarrow 0$ 12:     $sol\left[0\right]\left[i\right]\leftarrow -\infty$ 13:     $a\left[0\right]\left[i\right]\leftarrow 0$ 14: end for   Main 15: for $k\leftarrow 1$ to $2n-1$ do 16:     for all $1\le i,j\le n$ in parallel do 17:         if $control\left[i\right][j-1]=1$ then 18:            $column\left[i\right]\left[j\right]\leftarrow column[i-1]\left[j\right]+a\left[i\right]\left[j\right]$ 19:            $can{d}_1\leftarrow W\left[i\right][j-1]+column\left[i\right]\left[j\right]$ 20:            $can{d}_2\leftarrow W[i-1]\left[j\right]+a\left[i\right]\left[j\right]$ 21:            $can{d}_3\leftarrow old\_W\left[i\right]\left[j\right]+top\left[i\right]\left[j\right]$ 22:            $old\_W\left[i\right]\left[j\right]\leftarrow W\left[i\right]\left[j\right]$ 23:            $W\left[i\right]\left[j\right]\leftarrow \mathbb{MAX}\{can{d}_1,can{d}_2,can{d}_3\}$ 24:            $top\left[i\right]\left[j\right]\leftarrow top[i-1]\left[j\right]$ 25:            $sol\left[i\right]\left[j\right]\leftarrow \mathbb{MAX}\left\{sol\right[i-1\left]\right[j]$, $sol\left[i\right][j-1]$, $sol\left[i\right]\left[j\right]$, $W\left[i\right]\left[j\right]\}$ 26:            $control\left[i\right]\left[j\right]\leftarrow 1$ 27:         end if 28:     end for 29: end for

Note that the memory requirement for each cell in Algorithm 8 is constant. When we put W-shapes and N-shapes together, however, we need $O\left(n\right)$ space in each cell, as we will explain later on. We assume that all corresponding instructions in all cells are executed in parallel in a synchronized manner. We later make some comments regarding how synchronization is achieved in the actual implementation of the parallel algorithm.

We prove the correctness of Algorithm 8 in the framework of Hoare logic [11] based on a restricted form of that in Owicki and Gries [12]. The latter is too general to cover our problem. We keep the minimum extension of Hoare logic to our mesh architecture.

The meaning of Hoare’s triple $\left\{P\right\}S\left\{Q\right\}$ is that if P is true before program (segment) S, and if S halts, then Q is true after S stops. The typical loop invariant appears as that for a while-loop; “while B do S”. Here, S is a program, and B is a Boolean condition. If we can prove $\{P\wedge B\}S\left\{P\right\}$, we can conclude $\left\{P\right\}$ while B does $S\{P\wedge \sim B\}$, where ∼B is the negation of B. P is called the loop invariant, because P holds whenever the computer comes back to this point to evaluate the condition B. This is a time-wise invariant as the computer comes back to this point time-wise. We establish invariants in each cell. They are regarded as time-space invariants because the same conditions hold for all cells as computation proceeds. Those invariants have space indices i and j and time index k. Thus, our logical framework is a specialization of Owicki and Gries to indexed assertions.

The main assertions are given in the following. $scope[a,\dots ,b][c,\dots ,d]$ is the rectangular array portion from the top-left corner $(a,c)$ to the bottom-right corner $(b,d)$ where a candidate for the solution can be found.

At the end of the k-th iteration, the following holds in $cell(i,j)$:

For $1\le i\le n$ and $0\le j\le k$:

$$\begin{array}{cc}\hfill P_0:& control\left[i\right]\left[j\right]=1\hfill \\ \hfill P_1:& column\left[i\right]\left[j\right]\mathrm{is}\mathrm{the}\mathrm{column}\mathrm{sum}\mathrm{of}a[i+j-k,\dots ,i]\left[j\right],\hfill \\ & \mathrm{which}\mathrm{is}\mathrm{the}\mathrm{sum}\mathrm{of}\mathrm{the}j-\mathrm{th}\mathrm{column}\mathrm{of}\mathrm{array}a\mathrm{from}\mathrm{position}i+j-k\mathrm{to}\mathrm{position}i\hfill \\ \hfill P_2:& top\left[i\right]\left[j\right]=a[i+j-k]\left[j\right]\hfill \\ \hfill P_3:& W\left[i\right]\left[j\right]\mathrm{is}\mathrm{the}\mathrm{maximum}W\mathrm{anchored}\mathrm{at}\mathrm{column}a[i+j-k,\dots ,i]\left[j\right]\hfill \\ \hfill P_4:& old\_W\left[i\right]\left[j\right]\mathrm{is}\mathrm{the}\mathrm{old}\mathrm{value}\mathrm{of}W\left[i\right]\left[j\right]\hfill \\ \hfill P_5:& sol\left[i\right]\left[j\right]\mathrm{is}\mathrm{the}\mathrm{maximum}W\mathrm{in}scope[i+j-k,\dots ,i][1,\dots ,j]\hfill \end{array}$$

The above are combined to P, where:

$$P\iff (1\le i\le n\wedge 0\le j\le k\Rightarrow P_0\wedge P_1\wedge P_2\wedge P_3\wedge P_4\wedge P_5)\wedge (1\le i\le n\wedge k<j\le n\Rightarrow Q)$$

Here, Q states that variables in each cell keep the initial values. In the following descriptions, we omit the second portion Q of the above logical formula.

We can prove that $P_0\left(k\right)$ to $P_5\left(k\right)$ are all true for $k=0$ by checking the initialization. For each $P_0$ to $P_5$, we omit indices i and j. Using the time index k, we prove $\left\{P(k-1)\right\}cell(i,j)\left\{P_0\left(k\right)\right\}$ to $\left\{P(k-1)\right\}cell(i,j)\left\{P_5\left(k\right)\right\}$. We use the following rules of Hoare logic. Let $x_1\leftarrow y_1$ to $x_n\leftarrow y_n$ be assignment statements in $cell\left(1\right)$ to $cell\left(n\right)$ in general. There can be several in each cell. We use one for simplicity. The meaning of $y_i/x_i$ is that the occurrence of variable $x_i$ in Q is replaced by $y_i$. Parallel execution of $cell\left(1\right)$ to $cell\left(n\right)$ is shown by $\left[cell\right(1\left)\right|\left|cell\right(2)\dots |\left|cell\right(n\left)\right]$.

Parallel assignment rule:

$$\begin{array}{c}P\Rightarrow Q[y_1/x_1,y_2/x_2,\dots ,y_n/x_n],\hfill \\ \frac{\left\{Q[y_1/x_1,y_2/x_2,\dots ,y_n/x_n]\right\}cell\left(i\right)\left\{Q\right\}\mathbf{for}1\le i\le n}{\left\{P\right\}\left[cell\right(1\left)\right|\left|cell\right(2)\dots |\left|cell\right(n\left)\right]\left\{Q\right\}}\hfill \end{array}$$

Other programming constructs such as composition (semi-colon), if-then-else statement, etc., in sequential Hoare logic can be extended to the parallel versions. Those definitions are omitted, but the following rule for the if-then and for-loop for the sequential control structure, which controls a parallel program S from outside, is needed for our verification purpose.

Rule for if-then statement:

$$\frac{\{P\wedge B\}S\left\{Q\right\},P\wedge \neg B\Rightarrow Q}{\left\{P\right\}\mathbf{if}B\mathbf{then}S\left\{Q\right\}}$$

In our proof, $P(k-1)$ corresponds to P, $control\left[i\right][j-1]=1$ to B and $P\left(k\right)$ to Q.

Rule for the for-loop:

$$\frac{\left\{P\right(0\left)\right\},\left\{P\right(k-1\left)\right\}S\left\{P\right(k\left)\right\}}{\left\{P\right(0\left)\right\}\mathbf{for}k=1\mathbf{to}n\mathbf{do}S\left\{P\right(n\left)\right\}}$$

This P represents $P_0$ to $P_5$ in our program. S is the parallel program $\left[cell\right(1\left)\right|\left|cell\right(2)\dots |\left|cell\right(n\left)\right]$. Each $cell\left(i\right)$ has a few local variables and assignment statements. For an arbitrary array x, we regard $x\left[i\right]\left[j\right]$ as a local variable for $cell(i,j)$. A variable from the neighbour, $x[i-1]\left[j\right]$, for example, is imported from the upper neighbour. Updated variables are fetched in the next cycle. The proof for each $\left\{P\right(k-1\left)\right\}cell(i,j)\left\{P\right(k\left)\right\}$ for P is given in Appendix.

Theorem 1.

Algorithm 8 is correct. The result is obtained at $cell(n,n)$ in $2n-1$ steps.

Proof. 

From the Hoare logic rule for the for-loop, we have $P_5(2n-1)$ at the end.

$$\begin{array}{cc}\hfill P_5(2n-1)\mathrm{at}cell(n,n)& \iff sol\left[n\right]\left[n\right]\mathrm{is}\mathrm{the}\mathrm{maximum}\mathrm{sum}\mathrm{in}scope[n+n-2n+1,\dots ,n][1,\dots ,n]\hfill \\ & \iff sol\left[n\right]\left[n\right]\mathrm{is}\mathrm{the}\mathrm{maximum}\mathrm{sum}\mathrm{in}scope[1,\dots ,n][1,\dots ,n]\hfill \end{array}$$

☐

We used array $W[1..n][1..n]$ to compute the maximum W-shape. The value of $W\left[i\right]\left[j\right]$ at $cell(i,j)$ is ephemeral in the sense that its value changes as computation proceeds. That is, $W\left[i\right]\left[j\right]$ at time k holds the maximum W-shape anchored at column $a[i+j-k,\dots ,i]\left[j\right]$, and at the next step, it changes to that value of W anchored at column $a[i+j-k-1,\dots ,i]\left[j\right]$.

In order to combine W and N, we must memorize the maximum W and the maximum N for each anchor column. Thus, we need the three-dimensional array $store\_W$, as well as the array $store\_N$. Since in the final stage of $WN$ computation, the same anchor column is added from W and N, we need to subtract the sum of the anchor column. For that purpose, the sum of anchor column $a[k,\dots ,i]\left[j\right]$ is stored in $store\_column\left[i\right]\left[k\right]\left[j\right]$. Note that the computation of N goes from right to left.

In Algorithm 9, we provide the complete algorithm that combines W and N, where $sol$ is dropped, and the initialization is omitted. The computation of W and N takes $2n-1$ steps, and $WN$ takes n steps.

Selecting the maximum of $WN[1..n][1..n]$ can be done in parallel. Algorithm 10 and Figure 9a illustrate how to find the maximum in a 2D mesh in $2n$ steps, where the maximum can be retrieved at the bottom right corner. If we orchestrate the bidirectional data movement in each of four quarters of the mesh (Figure 9b), so that the maximum of each quarter can meet at the centre for the final selection, it can be done in $n+1$ steps. Therefore, Algorithm 9 takes total of $4n$ communication steps.

Algorithm 9 Combining W and N.

1: for $k\leftarrow 1$ to $2n-1$ do 2:     for all $1\le i,j\le n$ in parallel do 3:         if $control\left[i\right][j-1]=1$ then         ▹ Compute W from left to right 4:            $column\left[i\right]\left[j\right]\leftarrow column[i-1]\left[j\right]+a\left[i\right]\left[j\right]$ 5:            $can{d}_1\leftarrow W\left[i\right][j-1]+column\left[i\right]\left[j\right]$ 6:            $can{d}_2\leftarrow W[i-1]\left[j\right]+a\left[i\right]\left[j\right]$ 7:            $can{d}_3\leftarrow old\_W\left[i\right]\left[j\right]+top\left[i\right]\left[j\right]$ 8:            $old\_W\left[i\right]\left[j\right]\leftarrow W\left[i\right]\left[j\right]$ 9:            $W\left[i\right]\left[j\right]\leftarrow \mathbb{MAX}\{can{d}_1,can{d}_2,can{d}_3\}$ 10:            $store\_column\left[i\right][i+j-k]\left[j\right]\leftarrow column\left[i\right]\left[j\right]$ 11:            $store\_W\left[i\right][i+j-k]\left[j\right]\leftarrow W\left[i\right]\left[j\right]$ 12:            $top\left[i\right]\left[j\right]\leftarrow top[i-1]\left[j\right]$ 13:            $control\left[i\right]\left[j\right]\leftarrow 1$ 14:         end if 15:         if $control\left[i\right][j+1]=1$ then         ▹ Compute N from right to left 16:            Mirrored operation of lines 4–13 17:         end if 18:     end for 19: end for 20: for all $1\le i,j\le n$ in parallel do 21:     $WN\left[i\right]\left[j\right]\leftarrow -\infty$ 22:     for $k\leftarrow 1$ to i do 23:         $anchor\leftarrow store\_column\left[i\right]\left[k\right]\left[j\right]$ 24:         $WN\left[i\right]\left[j\right]\leftarrow \mathbb{MAX}\left\{WN\right[i\left]\right[j],store\_W[i\left]\right[k\left]\right[j]+store\_N[i\left]\right[k\left]\right[j]-anchor\}$ 25:     end for 26: end for 27: Find the maximum of $WN[1..n][1..n]$ by Algorithm 10

Algorithm 10 Find the maximum of $a[1..n][1..n]$ in $2n$ steps.

1: for all $1\le i,j\le n$ in parallel do 2:     $M\left[i\right]\left[j\right]\leftarrow a\left[i\right]\left[j\right]$ 3: end for 4: for $k\leftarrow 1$ to n do 5:     for all $1\le i,j\le n$ in parallel do 6:         $M\left[i\right]\left[j\right]\leftarrow \mathbb{MAX}\left\{M\right[i-1\left]\right[j],M[i\left]\right[j\left]\right\}$ 7:     end for                 ▹ Find the maximum in the same column 8: end for 9: for $k\leftarrow 1$ to n do 10:     for all j in parallel do 11:         $M\left[n\right]\left[j\right]\leftarrow \mathbb{MAX}\left\{M\right[n\left]\right[j-1],M[n\left]\right[j\left]\right\}$ 12:     end for                   ▹ Find the maximum in the n-th row 13: end for                         ▹$M\left[n\right]\left[n\right]$ is the maximum

6. Computation of the Boundary

So far, we computed the maximum sum of the $WN$-shape. For practical purposes, we sometimes need the shape itself. This problem was solved by Thaher [13] for the sequential algorithm (Algorithm 11). We show how the idea works in our parallel situation. For simplicity, we only show the boundary computation of the W-shape. We prepare the array $store\_d\left[i\right]\left[k\right]\left[j\right]$ to memorize the direction from which the algorithm took one of Case 1, Case 2 or Case 3. This part of enhancement in Algorithm 9 is shown below.

$$\begin{array}{c}\hfill \mathrm{Case}1:store\_d\left[i\right][i+j-k]\left[j\right]\leftarrow 1\\ \hfill \mathrm{Case}2:store\_d\left[i\right][i+j-k]\left[j\right]\leftarrow 2\\ \hfill \mathrm{Case}3:store\_d\left[i\right][i+j-k]\left[j\right]\leftarrow 3\end{array}$$

Let array $boundary\left[i\right]\left[j\right]=1$, if $(i,j)$ is on the boundary, and zero otherwise. Let $(i,k,j)$ denote the j-th column from position k to position i, to represent the anchor column of some W-shape. Suppose the anchor column of the maximum W-shape after Algorithm 9 is $(i_0,k_0,j_0)$. The following sequential algorithm goes backward from $(i_0,k_0,j_0)$ guided by the data structure $store\_d\left[i\right]\left[k\right]\left[j\right]$. The correctness of the algorithm is obvious. The time $O\left(n\right)$ can be shown as follows.

Algorithm 11 Computing the boundary.

1: $i\leftarrow i_0,j\leftarrow j_0,k\leftarrow k_0$ 2: while $j>0$ and $W\left[i\right]\left[k\right]\left[j\right]>0$ do 3:     $boundary\left[i\right]\left[j\right]\leftarrow 1$ 4:     $boundary\left[k\right]\left[j\right]\leftarrow 1$ 5:     $D\leftarrow store\_d\left[i\right]\left[k\right]\left[j\right]$ 6:     if D=1 then 7:         $j\leftarrow j-1$ 8:     else if D=2 then 9:         $i\leftarrow i-1$ 10:    else if D=3 then 11:        $k\leftarrow k+1$ 12:    end if 13: end while

The time is proportional to the number of changes on indices i, j and k. The index j can be reduced at most n times. For i and k, we observe $i-k$ decreases whenever i or k changes, resulting in $O\left(n\right)$ time for i and k. The $O\left(n\right)$ time for the boundary can be absorbed in $O\left(n\right)$ time of Algorithm 9. It will be easy to organize parallel computation of tracing back over array $store\_d\left[i\right]\left[k\right]\left[j\right]$ by the mesh architecture. For example, we can convert the array index j to a processor index j and use a one-dimensional mesh architecture as shown below (Algorithm 12).

This version has $O\left(n\right)$ time, which provides no gain time-wise, but the $O\left(n^3\right)$ space requirement on a single processor can be eased.

Algorithm 12 Computing the boundary; j-th cell.

1: receive $(i,k)$ 2: while $j>0$ and $W\left[i\right]\left[k\right]>0$ do 3:     $boundary\left[i\right]\left[j\right]\leftarrow 1$ 4:     $boundary\left[k\right]\left[j\right]\leftarrow 1$ 5:     $D\leftarrow store\_d\left[i\right]\left[k\right]$ 6:     if D=1 then 7:         send $(i,k)$-to $j-1$-th processor 8:     else if D=2 then 9:         $i\leftarrow i-1$ 10:    else if D=3 then 11:        $k\leftarrow k+1$ 12:    end if 13: end while Main 14: for $1\le j\le n$ in parallel do send $(i_0,k_0)$ to $j_0$-th processor 15: end for

7. Implementation

We implemented Algorithm 9 on the Blue Gene/P computer under the MPI/Parallel C program environment. There were many practical issues to be considered. We summarize just three issues here as representatives.

Firstly, we cannot assume that each cell knows its own position within the mesh array. Depending on the architecture, additional computation is required for each cell to gather this information. Within the MPI environment, we can let each $cell(i,j)$ know its position $(i,j)$ by the system call “MPI_Cart_coords()”.

The second issue is synchronization. We assumed the corresponding statements in all cells are executed in a synchronized manner. If we remove this assumption, that is if the execution proceeds in an asynchronous manner, the algorithm loses its correctness.

In MPI, “MPI_Send()”, “MPI_Recv()” and “MPI_Sendrecv()” functions are used to perform synchronous blocking communications [14]. As we call these functions in each step, no further mechanisms are necessary to ensure synchronization between cells as the function calls ensure that any given cell cannot progress one or more steps further than the rest. In other words, one cell may reach the end of the given step and tries to move onto the next step before others, but then, the cell must wait for the other cells to reach the same point due to the blocking nature of the MPI communication functions.

The third implementation issue is related to the number of available processors. As the number of processors is limited, for large n, we need to have what is called a coarse grain parallel computer. Suppose, for example, we are given a $(1024,1024)$ input array and only 16 processors are available. The input array is divided into sixteen $(256,256)$ sub-arrays, to which the sixteen processors are assigned. Let us call the sub-array for each processor its territory. Each processor simulates one step of Algorithm 8 sequentially. These simulation processes by sixteen processors are done in parallel. At the end of each simulation, the values in the registers on the right and bottom border are sent to the left and top borders of the right neighbour and the lower neighbour, respectively. The simulation of one step takes $O({(n/p)}^2)$ time, and $2n-1$ steps are carried out, meaning the computing time is $O(n^3/p^2)$ at the cost of $O\left(p^2\right)$ processors. When $p=1$, we hit the sequential complexity of $O\left(n^3\right)$. If $p=n$, we have the time complexity of Algorithm 8, which is $O\left(n\right)$.

While resolving the third implementation issue, we must again face the second issue of synchronization. For each processor to simulate the parallel computation in its own territory in a coarse-grained implementation, we must take extra care to ensure synchronization between simulated cells within each territory. Specifically, we must double the number of variables, that is we prepare variable $x_1$ for every variable x. Let us associate the space/time index, $(i,j,k)$ with each variable. Let us call $x(i,j,k)$ the current variable and the variable with indices different by one a neighbour variable. For example, $sol\left[i\right]\left[j\right]$ in the right-hand side of the assignment statement is a time-wise neighbour, and that at the left-hand side is a current variable. Furthermore, $column[i-1]\left[j\right]$ in the right-hand side is a neighbour variable space-wise and time-wise, and so on. If x is a current variable, change it to $x_1$. If it is a variable of a neighbour, keep it as it is. Let us call the modified program $P_1$. Now, we define “update” to be the set of assignment statements of the form $x\leftarrow x_1$.

Example 1.

Let P be a one-dimensional mesh program given by Algorithm 13, which shifts array x by one place. Let us suppose $x\left[0\right]=0$ and $x\left[i\right]$ are already given.

In Algorithm 13, $x\left[i\right]$ is the current variable, and $x[i-1]$ is a neighbour variable space-wise and time-wise. An asynchronous computer can make all values zero. For the intended outcome, we perform $P_1$, given by Algorithm 14, which includes “synchronize” and “update”.

Algorithm 13 Program P.

1: for all i in parallel do 2:     $x\left[i\right]\leftarrow x[i-1]$ 3: end for

Algorithm 14 Program $P_1$.

1: for all i in parallel do 2:     $x_1\left[i\right]\leftarrow x[i-1]$ 3: end for 4: synchronize 5: for all i in parallel do 6:     $x\left[i\right]\leftarrow x_1\left[i\right]$ /* update */ 7: end for

For our mesh algorithm, Algorithm 8, omitting the initialization part, we make the program of the form that is given by Algorithm 15.

Algorithm 15 Synchronization for coarse grain.

1: for $k\leftarrow 1$ to $2n-1$ do 2:     $P_1$ 3:     synchronize 4:     update 5: end for

Algorithm 9 was executed on Blue Gene/P with up to 1024 cores. For the software side, the programming environment of MPI and the parallel C compiler, mpixlc, were used with Optimization Level 5. The results are shown in

Table 1. Time measurements on BG/P(seconds).

	${\mathit{p}}^2$ = 4	16	64	256	1024
n = 64	0.1069	0.0329	0.0258	0.0251	0.0272
128	0.8435	0.2298	0.0989	0.1253	0.1252
256	7.2762	1.7506	0.4783	0.3783	0.6088
512	64.5455	15.8192	3.6038	1.3141	2.9883
1024	332.8847	137.6626	33.9193	7.7712	7.9303
2048	x	x	188.9598	87.5153	43.5064

. Arrays of size $(n,n)$ containing random integers were tested. The times for generating uniformly-distributed random numbers, as well as loading the data onto each processor were not included in the time measurement. On Blue Gene/P, the mesh architecture can be configured into a 2D mesh or 3D mesh. The time taken to configure the mesh into a 2D mesh array for our algorithm was included in the time measurement. As we can see from the table, for small n, the configuration time dominates, and increasing the number of processors does not result in a decrease in the execution time. As the size of the input array increases, however, we can see noticeable improvements in the execution times with a larger number of processors.

Note that we were unable to execute the program under certain configurations as shown by `x’ in Table 1. This was due to memory requirements. With large n and small p, each processor must simulate a very large territory. With each cell in the territory requiring $O\left(n\right)$ memory to store the maximal values of W and N, the memory requirements can become prohibitive. This highlights the fact that parallelization is required not only to reduce the execution time, but also to handle larger input data.

8. Lower Bound

Algorithm 8 for W is not very efficient, as cells to the right are idling at the early stage. Suppose we are given an input array with the value a in the top-left cell and b in the bottom-right cell, while all other value are $-\infty$. Obviously the solution is a if $a>b$, and b otherwise. The values a and b need to meet somewhere. It is easy to see that the earliest possibility is at time $n-1$. Our algorithm for W takes $2n-1$ steps, meaning there is still a gap of n steps with the lower bound. This is a sharp contrast with the mesh algorithm for MSA (Algorithm 4) that completes in $n-1$ steps. The role of Algorithm 8 is to establish an $O\left(n\right)$ time for the MCS problem on the mesh architecture.

9. Concluding Remarks

We gave an $O\left(n\right)$ time parallel algorithm for solving the MSA and MCS problem with $n^2$ processors and a formal proof for the algorithm to compute the W-shape, a part of the MCS problem. The formal proof for the N-shape can be given in a similar way. The formal proof not only ensures correctness, but also clarifies what is actually going on inside the algorithm.

The formal proof was simplified by assuming synchronization. The asynchronous version with (synchronize, update) in Section 7 would require about twice as much complexity for verification, since we double the number of variables. Once the correctness of the synchronized version is established, that of the asynchronous version will be acceptable without further verification.

It is open whether there is a mesh algorithm for the MCS problem with the $O\left(1\right)$ memory requirement in each cell. Mesh algorithms are inherently easy to implement on an FPGA and, thus, can be considered for practical applications.

multiple