Superpixels with Content-Awareness via a Two-Stage Generation Framework

Abstract

The superpixel usually serves as a region-level feature in various image processing tasks, and is known for segmentation accuracy, spatial compactness and running efficiency. However, since these properties are intrinsically incompatible, there is still a compromise within the overall performance of existing superpixel algorithms. In this work, the property constraint in superpixels is relaxed by in-depth understanding of the image content, and a novel two-stage superpixel generation framework is proposed to produce content-aware superpixels. In the global processing stage, a diffusion-based online average clustering framework is introduced to efficiently aggregate image pixels into multiple superpixel candidates according to color and spatial information. During this process, a centroid relocation strategy is established to dynamically guide the region updating. According to the area feature in manifold space, several superpixel centroids are then split or merged to optimize the regional representation of image content. Subsequently, local updating is adopted on pixels in those superpixel regions to further improve the performance. As a result, the dynamic centroid relocating strategy offers online averaging clustering the property of content awareness through coarse-to-fine label updating. Extensive experiments verify that the produced superpixels achieve desirable and comprehensive performance on boundary adherence, visual satisfactory and time consumption. The quantitative results are on par with existing state-of-the-art algorithms in terms with several common property metrics.

1. Introduction

In the field of image processing, the concept of superpixels is generalized to semantically segment a visual image into several simple connected sub-regions [1]. A superpixel is intuitively a collection of spatially connected pixels with similar low-level visual features such as color and texture, which could represent a mass of image entities [2]. On this basis, superpixel segmentation has gradually become a popular preprocessing tool for various advanced computer vision tasks to boost running efficiency. The emerging superpixel algorithms have found applications in diverse fields, including image classification [3,4], semantic segmentation [5,6], and object detection and tracking [7,8].

Since their inception in 2003, the pursuit of superpixels with sensational performance has remained a focal point in this field. In recent years, an increasing number of superpixel algorithms have been proposed to improve on former achievements [9,10,11,12]. In general, a good superpixel generation algorithm has properties including boundary accuracy, visual satisfactory and running efficiency. These requirements also lead to an essential difference from conventional image over-segmentation or contour detection, whose practical significance can be reflected in the following three aspects [13]:

• Boundary accuracy. When employed as a region-level feature for image segmentation or edge detection, it evaluates the effectiveness of superpixels in delineating image object boundaries and their consistency with the ground truth;
• Feature quality. For some applications such as image reconstruction and data compression, it is important to preserve regional homogeneity, spatial relationships and other information pertaining to internal objects within an image;
• Running efficiency. As an advantaged tool for video and image preprocessing, superpixel segmentation plays a pivotal role in computer vision applications that necessitate real-time performance.

Nevertheless, these properties still suffer from a certain degree of incompatibility, resulting in a rigorous constraint on the properties observed in the generated superpixels. In practice, this can be delineated as two sub-problems based on their manifestations.

Accuracy and quality: Real-world scenarios often exhibit diverse objects and textured backgrounds, leading to a common compromise between segmentation accuracy and feature quality in most superpixel methods [14]. For example, it is usually difficult for superpixels with color-based measurement to maintain a balanced trade-off between boundary adherence and region homogeneity, especially in low-contrast regions [15]. In addition, a greater compactness for superpixels sometimes deteriorates the accuracy of region representation despite promoting the visual satisfactory and spatial topology [16]. Furthermore, since regular visual information is rarely observed, the criterion of size uniformity is hard to follow strictly. A typical instance of this incompatibility is twig and sinuous object representation, which becomes an intractable problem for most algorithms except for some specialized optimizations [17,18].

Performance and efficiency: Most methods mainly create superpixels by solving a cost function established by global and local correlations of several low-level features [19]. The generation process is then recast as a minimization problem. Since it is time-consuming to find the globally optimal solution, a sub-optimal substitute is usually utilized to find the best compromise. Among them are clustering-based approaches, which usually adopt an empirical iteration time as the expiration. Nevertheless, it might be insufficient for detailed regions to converge accurately. Moreover, some dispensable strategy in structured superpixel generation frameworks might introduce additional time consumption, such as initialization and post-processing optimization.

Given the diverse performance of various superpixel generation algorithms in the literature, it remains a major challenge to seek synergistic optimization rather than property compromise. On the other hand, several instructive works have achieved state-of-the-art performance with respect to specific aspects. In practical applications, the combination of their strengths might scale new heights. Building upon this premise, this paper proposes a novel approach termed superpixels with Content-Awareness via a Two-Stage generation framework (CATS). It combines the clustering pattern of [20] with the density evaluation of [21], which enhances both pixel classification precision and image content sensitivity. As a result, it can produce superpixels with better boundary adherence and visual satisfaction while maintaining operational efficiency.

In summary, the main contributions of this work are as follows:

• According to the area in manifold space, a region-density-based clustering centroid relocating strategy is proposed to adjust the spatial distribution of superpixels;
• By using redistributed clustering centers, a coarse-to-fine implementation of online average clustering framework is designed for improving the feature performance;
• The synergistic CATS superpixel feature demonstrates comparable performance in terms of segmentation accuracy, spatial compactness, and operational efficiency.

The subsequent sections are organized as follows. Section 2 reviews several state-of-the-art (SOTA) superpixel algorithms. In Section 3, the proposed CATS superpixel generation framework is elaborated on in detail. Section 4 presents both qualitative and quantitative analyses. Concluding remarks and future perspectives are included in Section 5.

2. Related Work

Over the past two decades, the field has witnessed the emergence of numerous approaches for generating superpixels. Several reviews have presented comprehensive overviews of the previous works and categorized them based on mathematical principles or low-level features [22]. In this paper, the most SOTA superpixel algorithms are simply classified into two categories, iteration-demand and iteration-free. In the following review, only the closely related literature is discussed in detail.

2.1. Iteration-Demand Superpixels

Iteration-demand superpixels usually establish an energy function by the sum of pixel-cluster correlation or intra-region heterogeneity. In this way, the superpixel generation process is recast as a minimization problem that can be iteratively optimized [23].

Simple Linear Iterative Clustering (SLIC) [24] is widely recognized as a seminal work in this field, and its heuristic framework has garnered significant acclaim from subsequent researchers. The generation of SLIC superpixels is essentially an unsupervised classification process based on pixel clustering. Specifically, it restricts the conventional k-means algorithm into local context to reduce the calculation of Centroidal Voronoi Tessellation (CVT) in a 5-dimensional fusion feature space. The correlation between a pixel and a cluster centroid is measured by a normalized distance in both 3-dimensional CIELAB color space and 2-dimensional Euclidean space. Based on the measurement with a spatial constraint term, SLIC superpixels exhibit satisfactory color homogeneity while maintaining pleasant spatial compactness. Due to the exceptional overall performance of SLIC, various enhancements encompassing feature substitution, structure optimization and clustering acceleration have been made to this framework in subsequent studies.

Manifold SLIC (MSLIC) [25] and Minimum Barrier Superpixel (MBS) [26] are two representative variants with emphasis on both feature and structure innovation. MSLIC maps the 5-dimensional fusion feature space to a 3-dimensional manifold space, and then computes a Restricted CVT on the manifold surface. By utilizing the property that area elements in this space can effectively measure image content density, the inverse projection could induce content-sensitive superpixel segmentation results. Compared with the conventional SLIC, MSLIC superpixels better preserve object boundaries in regions with detailed image content while maintaining regular smoothness with desirable compactness. MBS proposes a novel feature correlation measurement for pixel classification within the iterative clustering framework. The proposed Compact-Aware Minimum Barrier Distance (CA-MBD) can not only control the compactness continuously, but maintains a balanced trade-off with accuracy. In addition, an efficient raster-scanning approach is introduced to guide the computation of CA-MBD, which guarantees the overall running efficiency.

Boundary-Aware Superpixel Segmentation (BASS) [27] and Content-Adaptive Superpixel (CAS) [28] introduce additional image cues along with the most frequently used color and spatial features. BASS incorporates a grayscale-weighted geodesic distance term to measure the correlation between pixels and superpixel clusters. In this way, BASS enables smaller superpixels to focus on informative areas while larger ones cover evenly distributed regions. In comparison to the SLIC algorithm, it generates superpixels with higher regional homogeneity and boundary accuracy. CAS incorporates four low-level features to model global superpixels, including color, texture, contour gradient, and spatial distance. An objective function is formulated based on inter-superpixel and intra-superpixel variances, which is then iteratively optimized through adaptive weight updates. During this process, the feature weights with weaker discrimination capabilities are updated by clustering results that demonstrate stronger discrimination, so as to accurately classify pixels of different image objects. Experimental results report that it exhibits exceptional intra-superpixel homogeneity and compactness in distribution with adaptability towards variations in image content.

There are others in this category that focus on speeding up the clustering efficiency. In the work of [29], a new approach termed Iterative Boundaries implicit Identification Superpixels (IBIS) is proposed to further reduce the complexity of SLIC-like methods. This can efficiently identify the coherence of the local region via a coarse-to-fine block-wise subdivision, and then update only a fraction of the pixels in the heterogeneous region in subsequent iterations. Consequently, the amount of computation for global updating in each iteration enters a steep decline. Compared with the conventional SLIC, it runs over 8 times faster than the conventional SLIC while performing with a matched segmentation accuracy. Fast Linear Iterative Clustering (FLIC) [30] superpixels incorporate feature continuity between neighboring pixels in real-world scenarios, facilitating consistent label assignment during the clustering process. By introducing an active search strategy based on adjacent pixel correlations, FLIC relaxes the constraints on k-means clustering search range in SLIC. This modification enables FLIC superpixels to simultaneously perform label assignment and cluster updates while significantly reducing computation time required for pixel-cluster similarity. In practical applications, this framework achieves rapid convergence of all superpixel clusters with just 2–3 global iterations.

2.2. Iteration-Free Superpixels

Several algorithms can generate superpixels without clustering, which primarily relies on a one-pass framework capable of achieving global optimal or suboptimal results [31]. They merely (potentially) necessitate making multiple adjustments to regional pixels or candidate superpixels and obviate multiple global updates.

Simple Non-Iterative Clustering (SNIC) [20] represents a significant improvement over SLIC, offering a more comprehensive approach. It employs a priority queue to efficiently inspect neighboring elements for pixel-cluster matching, which is manifested as a diffusion-based region growing process. Once a pixel is assigned with a definite label, it would be excluded in the subsequent operations. Compared with SLIC, SNIC effectively reduces the redundant computation during label updating in overlapping regions. Moreover, all superpixels are generated by absorbing neighboring pixels from the corresponding seed via a region growing pattern. Therefore, they can maintain spatial connectivity with uniformly distributed labels without the need for merging isolated partitions.

In the work of [32], Density-Based Spatial Clustering of Applications with Noise (DBSCAN) is introduced to optimize the clustering framework. In contrast to the k-means method, DBSCAN inspects all pixels only once without iterative execution. The proposed method adopts a split-and-merge scheme to generate superpixels. It first aggregates pixels into candidate superpixels by DBSCAN, and then merges the small clusters into adjacent homogenous superpixels to strengthen the spatial connectivity.

Ultra-fast Superpixel Extraction via Quantization (USEQ) [33] has been proposed for the rapid extraction of superpixels. Initially, spatial and color quantization are employed to approximately reconstruct the input image. Subsequently, the misclassified pixels are adjusted to their most visually and spatially similar superpixels using maximum a posteriori (MAP) estimation. In follow-up work, an adaptive sampling strategy was introduced to further enhance the content sensitivity of USEQ superpixels to the distribution of image content [34]. This strategy selects one or more superpixel candidates from each spatial quantization region based on the color variations, which enables the compactness to adapt to the differences in uniform or cluttered regions

The Dynamic Random Walk (DRW) [18] method maps the input image to an undirected graph model using graph theory and the Random Walk (RW) [35] model introduces a novel dynamic node to the conventional RW framework, effectively reducing redundant calculations by constraining the walking range. To address the issue of seed-lacking, the energy function of the random walk is redefined. Moreover, it incorporates first arrival probabilities between node pairs to mitigate interference across partitions. Consequently, DRW superpixels exhibit enhanced boundary adherence with linear time complexity.

Watershed Transform (WT) [36] is an efficient and instructive over-segmentation approach rooted in topological theory that can be described as a flooding process. It first identifies the minima of gradient images and subsequently submerges the corresponding pixels. To further control the amount and regularity of watershed partitions, WaterPixels (WP) [37] proposes a spatially regularized gradient to achieve a tunable trade-off between superpixel regularity and adherence to object boundaries. Essentially, it forms a joint gradient and spatial distance measurement similar to the feature correlation calculation in SLIC. More recently, in the work of [38], a two-phase implementation has been introduced in WT to further improve the performance. It designs different measurement for boundary pixels by two separate criteria that focus on the color homogeneity and shape regularity, respectively. As reported in the experiments, this approach could ameliorate the inter-inhibition between boundary adherence and spatial compactness.

3. Proposed Method

In this section, the proposed CATS framework is introduced in detail; the workflow is visualized in Figure 1 in advance for a better illustration. To improve the segmentation quality while maintaining the computational efficiency, the proposed approach performs superpixel extraction in a coarse-to-fine manner. A global online averaging clustering is first performed on the image plane to produce superpixel candidates and quantify the local content density in Section 3.1. From this foundation, an adaptive centroid relocation strategy is presented in Section 3.2 to dynamically split several candidates with dense content, as well as merge partitions with small areas. Subsequently, Section 3.3 shows the pseudocode of synthetic CATS implementation.

3.1. Superpixel Candidate Generation

The online averaging clustering efficiently aggregates pixels in a non-iterative manner. Substantially, it employs a greedy algorithm and utilizes a priority queue to assign unique labels to all pixels. In this part, the implementation of online averaging clustering in SNIC is followed to generate superpixel candidates.

3.1.1. Initialization and Seeding

Given an image $I={\left\{p_i\right\}}_{i=1}^N$ with $N$ pixels, let us denote by $P\left(p_i\right)=\left[x\left(p_i\right),y\left(p_i\right)\right]$ and $C\left(p_i\right)=\left[l\left(p_i\right),a\left(p_i\right),b\left(p_i\right)\right]$ the 2-dimensional Euclidean coordinate and 3-channel CIELAB color of each pixel $p_i\in I$, respectively. In SNIC, the correlation of a pair of pixels $p_i$ and $p_j$ is measured by a normalized Euclidean distance $D\left(p_i,p_j\right)$ in a joint color and spatial space ${ℝ}^5$ as follows

$$D\left(p_i,p_j\right)=\sqrt{{\left({\displaystyle \frac{D_{color}\left(p_i,p_j\right)}{N_c}}\right)}^2+{\left({\displaystyle \frac{D_{spatial}\left(p_i,p_j\right)}{N_s}}\right)}^2}$$

(1)

where $N_c$ and $N_s$ are two pre-defined factors to control the influence of color and position on $D\left(p_i,p_j\right)$, and

$$D_{color}\left(p_i,p_j\right)=\sqrt{{\left(l\left(p_i\right)-l\left(p_j\right)\right)}^2+{\left(a\left(p_i\right)-a\left(p_j\right)\right)}^2+{\left(b\left(p_i\right)-b\left(p_j\right)\right)}^2}$$

(2)

$$D_{spatial}\left(p_i,p_j\right)=\sqrt{{\left(x\left(p_i\right)-x\left(p_j\right)\right)}^2+{\left(y\left(p_i\right)-y\left(p_j\right)\right)}^2}$$

(3)

First, a zeroed label map $L={\left\{L\left(p_i\right)\right\}}_{i=1}^N$ is introduced to record the clustering result, whose size is identical to the input image $I$. Meanwhile, a priority queue $Q$ is implemented by a small-root binary tree, which still pops up the top-most element with the minimal key value when it is not empty. Next, a set of seeds ${\left\{p_k^{\mathrm{s}}\right\}}_{k=1}^K$ is acquired by uniformly sampling $K$ pixels on $I$; each seed is centered at a grid cell with the step of $S=\sqrt{N/K}$. The n-th evenly distributed seed $p_n^{\mathrm{s}}$ is assigned with a unique label (i.e., $L\left(p_n^{\mathrm{s}}\right)=n$), which serves as the incipient centroid of the n-th cluster region ${\Re }_n$. All seeds are then pushed onto $Q$.

3.1.2. Clustering and Updating

Each unlabeled adjacent pixel $p^{\mathrm{a}\left|\mathrm{g}\right.}$ to the growth point $p^{\mathrm{g}}$ of the current sprouting cluster ${\Re }^{\mathrm{g}}$ is inspected to calculate the pixel-cluster correlation $D\left(p^{\mathrm{a}\left|\mathrm{g}\right.},c^{\mathrm{g}}\right)$, which is measured by Equation (1). $c^{\mathrm{g}}$ represents the centroid of ${\Re }^{\mathrm{g}}$. Then $p^{\mathrm{a}\left|\mathrm{g}\right.}$ is pushed onto $Q$, and $D\left(p^{\mathrm{a}\left|\mathrm{g}\right.},c^{\mathrm{g}}\right)$ acts as the key value for heap sorting. It is worth noting that all seeds are initialized with a zero key value. In this case, they could become the first growth points within the corresponding cluster regions, respectively.

The top-most element $p^{\mathrm{t}\left|\mathrm{g}\right.}$ (one of $p^{\mathrm{a}\left|\mathrm{g}\right.}$ that holds the minimal key value in $Q$) is assigned with an identical label to its corresponding growth point $p^{\mathrm{g}}$ after it is popped, i.e., $L\left(p^{\mathrm{t}\left|\mathrm{g}\right.}\right)=L\left(p^{\mathrm{g}}\right)$. The current sprouting cluster ${\Re }^{\mathrm{g}}$ that absorbs $p^{\mathrm{t}\left|\mathrm{g}\right.}$ as a new member is then updated by

$$C\left({\Re }^{\mathrm{g}}\right)={\displaystyle \frac{C\left(c^{\mathrm{g}}\right)+C\left(p^{\mathrm{t}\left|\mathrm{g}\right.}\right)}{\left|{\Re }^{\mathrm{g}}\right|+1}},P\left({\Re }^{\mathrm{g}}\right)={\displaystyle \frac{P\left(c^{\mathrm{g}}\right)+P\left(p^{\mathrm{t}\left|\mathrm{g}\right.}\right)}{\left|{\Re }^{\mathrm{g}}\right|+1}}$$

(4)

where $\left|{\Re }^{\mathrm{g}}\right|$ indicates the member amount in ${\Re }^{\mathrm{g}}$. Thereafter, $p^{\mathrm{t}\left|\mathrm{g}\right.}$ replaces $p^{\mathrm{g}}$ as the latest growth point of ${\Re }^{\mathrm{g}}$ to inspect its unlabeled neighbor pixels in the next loop.

The clustering and updating are repeated until the priority queue $Q$ is empty and there are no pixels in the image with unassigned labels. Finally, it outputs the updated label map as the clustering result. Notice that an unlabeled pixel does not necessarily result in immediate labeling after being inspected. In practice, it might be inspected by different growth points, and the final label is determined by the cluster that is most relevant to it.

3.2. Centroid Relocation Strategy

The aforementioned implementation of superpixel candidate generation employs a grid-level seeding initialization, which partially ensures the size regularity and uniform distribution of superpixels. Nevertheless, due to the inconsistent density distribution of information in real-world images, superpixel generation relying solely on this process inevitably leads to mistaken label classification in several candidate regions. In other words, the performance of online averaging clustering is sensitive to the placement of initial seeds. To address this issue, a centroid relocation strategy is proposed based on the content density. This can evaluate and adjust the cluster centroid (position and amount) to guide the re-generation of some candidate superpixels.

3.2.1. Area of Manifold Surface

As illustrated in Figure 2, the key to perceiving the image content density in MSLIC is to transform the image plane into a 2-dimensional manifold space $\mathcal{M}$ through a stretching map $\Phi :I\to {ℝ}^5$. In this way, the feature of a pixel $p_i\in \mathcal{M}$ is described as $\Phi \left(p_i\right)=\left({\lambda }_{\mathrm{s}}P\left(p_i\right),{\lambda }_{\mathrm{c}}C\left(p_i\right)\right)$, where ${\lambda }_s=1/N_s$ and ${\lambda }_c=1/N_c$.

The area feature is very suitable for indicating the image content density on $\mathcal{M}$, which is further verified by the follow-up research of MSLIC [21]. Specifically, as shown in Figure 2, given a pixel $p_i\in I$ on the image plane, let us denote by ${\square }_p$ a unit square centered at $P\left(p_i\right)$ (red box filled by yellow), whose vertexes are notated as $p_i^1$, $p_i^2$, $p_i^3$ and $p_i^4$, respectively. Meanwhile, each vertex point holds the mean value of color and spatial feature of $p_i$ and its three 8-neighboring pixels. In this case, the area of ${\square }_p$ can be computed by

$$Area\left({\square }_p\right)=Area\left(\Delta p_i^1{p}_i^3{p}_i^4\right)+Area\left(\Delta p_i^1{p}_i^2{p}_i^4\right)$$

(5)

where

$$Area\left(\Delta p_i^1{p}_i^3{p}_i^4\right)\approx {\displaystyle \frac{1}{2}}‖\overrightarrow{\Phi \left(p_i^3\right)\Phi \left(p_i^1\right)}‖\cdot ‖\overrightarrow{\Phi \left(p_i^3\right)\Phi \left(p_i^4\right)}‖\cdot \sin \theta$$

(6)

In Equation (6), $\sin \theta$ measures the sine value between vector $\overrightarrow{\Phi \left(p_i^3\right)\Phi \left(p_i^1\right)}$ and $\overrightarrow{\Phi \left(p_i^3\right)\Phi \left(p_i^4\right)}$. $‖\overrightarrow{\Phi \left(p_i^3\right)\Phi \left(p_i^1\right)}‖$ represents the distance between $\Phi \left(p_i^3\right)$ and $\Phi \left(p_i^1\right)$ on $\mathcal{M}$, which is also calculated by Equation (1). Similar calculations can be done for $‖\overrightarrow{\Phi \left(p_i^3\right)\Phi \left(p_i^4\right)}‖$.

3.2.2. Splitting and Merging

Since the conventional SNIC lacks the ability of global updating, the generated superpixels are sensitive to the placement of initial centroids (seeds). To tackle this issue, a straightforward idea is to relocate the centroids, including perturbing the position of existing centroids and emerging new seeds for clustering. As shown in Figure 1e, the refined distribution of cluster centroids becomes more relevant to the local density, i.e., detailed regions acquire more centroids while sparse regions maintain the convergence.

In Figure 3, the area of each pixel on $\mathcal{M}$ can be calculated by Equation (5). Consequently, the total area of the manifold surface $Area(\mathcal{M})$ is obtained by summing up all pixels

$$Area\left(\mathcal{M}\right)={\displaystyle {\sum }_{i=1}^{N}Area\left(\Phi \left(\square p_i\right)\right)}$$

(7)

Assuming that the expected number of superpixels is $K$, the average area of all candidate superpixels on $\mathcal{M}$ is

$$\overline{Area\left(\Phi \left(\Re \right)\right)}={\displaystyle \frac{Area(\mathcal{M})}{K}},\hspace{1em}i=1,2,3,\cdot \cdot \cdot ,K$$

(8)

In fact, the area of k-th candidate superpixel ${\Re }_k$ on $\mathcal{M}$ is

$$Area\left(\Phi \left({\Re }_k\right)\right)={\displaystyle {\sum }_{p_i\in {\Re }_k}Area\left(\Phi \left(\square p_i\right)\right)}$$

(9)

The ratio of area on $\mathcal{M}$ can be notated as a scaling factor ${\lambda }_k$ to judge the splitting or merging operation, which is mandatory in the subsequent process.

$${\lambda }_k={\displaystyle \frac{Area\left(\mathcal{M}\right)}{Area\left(\Phi \left({\Re }_k\right)\right)}}$$

(10)

Specifically, when the area of a candidate superpixel exceeds twice that of an average superpixel (i.e., $K\ge 2{\lambda }_k$), its regional content density is identified as high. Therefore, the region corresponding to ${\Re }_k$ requires more superpixels with smaller sizes to adhere to complex boundaries while maintain spatial compactness. To this end, a primary step is to resample more seeds in this region.

The splitting implementation in [25] is partly learnt in this work. As for the centroid $c_k$ of superpixel candidate ${\Re }_k$, it is replaced by four symmetrically sampled seeds, $c_k^1$, $c_k^2$, $c_k^3$ and $c_k^1$ positioned at

$$\begin{array}{l}P\left(c_k^1\right)=\left(x\left(c_k\right)-{\displaystyle \frac{{\lambda }_k}{K}}S,y\left(c_k\right)+{\displaystyle \frac{{\lambda }_k}{K}}S\right)\\ P\left(c_k^2\right)=\left(x\left(c_k\right)-{\displaystyle \frac{{\lambda }_k}{K}}S,y\left(c_k\right)-{\displaystyle \frac{{\lambda }_k}{K}}S\right)\\ P\left(c_k^3\right)=\left(x\left(c_k\right)-{\displaystyle \frac{{\lambda }_k}{K}}S,y\left(c_k\right)-{\displaystyle \frac{{\lambda }_k}{K}}S\right)\\ P\left(c_k^4\right)=\left(x\left(c_k\right)+{\displaystyle \frac{{\lambda }_k}{K}}S,y\left(c_k\right)+{\displaystyle \frac{{\lambda }_k}{K}}S\right)\end{array}$$

(11)

In addition, the initial color information of these seeds is the same as the corresponding pixels on $I$.

There is a potential increase in the number of candidate superpixels after the aforementioned splitting operation, which might lead to a higher complexity for subsequent region-based information processing. Consequently, it becomes imperative to consider merging neighboring regions that exhibit low content density after global region splitting. In contrast to the conventional MSLIC, which incorporates intricate merging conditions and outputs only one seed for subsequent region iteration, a region-level merging approach is employed to directly update candidate superpixels. The procedures are outlined as follows.

First, a region-based undirected graph $G=\left(V,E\right)$ is established to depict the spatial relationship of all superpixel candidates [39]. A node vertex $v_k\in V$ represents a cluster centroid $c_k$, and the weight ${\omega }_{ij}$ of an edge $e_k\in E$ that connects a node pair $\left(v_i,v_j\right)$ depicts the inter-centroid color correlation

$${\omega }_{ij}=\left\{\begin{array}{c}{D}_{color}\left(v_i,v_j\right)\hspace{1em}\mathrm{if}\hspace{0.17em}{\Re }_i|{\Re }_j\\ 0\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{otherwise}\end{array}\right.$$

(12)

where ${\Re }_i|{\Re }_j$ means that the corresponding clusters of $c_i$ and $c_j$ are spatially adjacent. In this way, the value of ${\omega }_{ij}$ can effectively indicate the color homogeneity between cluster ${\Re }_i$ and ${\Re }_j$, which is calculated by Equation (2).

Next, the merging operation commences from the global minimum weight, if the corresponding clusters ${\omega }_m$ and ${\omega }_n$ differ greatly in areas (i.e., the ratio $\theta$ of the larger area compared to the smaller one is set to 4) the smaller superpixel is merged into the larger counterpart. Otherwise ($\theta <4$), the subsequent weight is selected in a descending order for processing.

$$\theta ={\displaystyle \frac{Area\left(\Phi \left({\Re }_m\right)\right)}{Area\left(\Phi \left({\Re }_n\right)\right)}},Area\left(\Phi \left({\Re }_m\right)\right)>Area\left(\Phi \left({\Re }_n\right)\right)$$

(13)

Obviously, this merging process effectively mitigates the size disparities among global superpixels while preserving regional homogeneity. Meanwhile, it can be ceased at any time, which allows for precise control over the final count of superpixels.

3.3. Integrated CATS Framework

As shown in Algorithm 1, the integrated CATS superpixel generation framework is raised by incorporating the area feature indicator and splitting/merging strategy into online average clustering. The modular structure is concise for realization based on the implementation of standard SNIC and MSLIC.

Algorithm 1 CATS superpixel generation framework

Input: Source image $I={\left\{p_i\right\}}_{i=1}^N$, the preset superpixel number $K$.

Output: Label map $L={\left\{L\left(p_i\right)\right\}}_{i=1}^N$ of $I$.

Initialize cluster seeds ${\left\{p_k^{\mathrm{s}}\right\}}_{k=1}^K$ by grid sampling with step $S=\sqrt{N/K}$.

Initialize $L\left(p_i\right)=0$ for each pixel $p_i\in I$.

Initialize a priority queue $Q$ with a small root.

for each seed $p_k^{\mathrm{s}}$ in ${\left\{p_k^{\mathrm{s}}\right\}}_{k=1}^K$ do

Create and push an element $e\left(p_k^{\mathrm{s}}\right)=\left[P\left(p_k^{\mathrm{s}}\right),C\left(p_k^{\mathrm{s}}\right),k,0\right]$ onto $Q$.

end for

while Q is not empty do

Pop the top-most element $\widehat{e}=e\left(p^{\mathrm{t}\left|\mathrm{g}\right.}\right)$ from $Q$.

if $L\left(p^{\mathrm{t}\left|\mathrm{g}\right.}\right)=0$ then

Set $L\left(p^{\mathrm{t}\left|\mathrm{g}\right.}\right)=L\left(p^{\mathrm{g}}\right)$.

Update ${\Re }^{\mathrm{g}}$ by Equation (4).

Compute ${\lambda }_k$ by Equation (10).

if $K\ge 2{\lambda }_k$ then

Compute seed set $\Omega$ by Algorithm 2.

Create and push the elements in $\Omega$ onto $Q$.

else

for each 4-neighboring pixel $p^{\mathrm{a}\left|\mathrm{g}\right.}$ of $p^{\mathrm{g}}$ do

if $L\left(p^{\mathrm{a}\left|\mathrm{g}\right.}\right)=0$ then

Create and push an element $e\left(p^{\mathrm{a}\left|\mathrm{g}\right.}\right)=\left[P\left(p^{\mathrm{a}\left|\mathrm{g}\right.}\right),C\left(p^{\mathrm{a}\left|\mathrm{g}\right.}\right),L\left(p^{\mathrm{g}}\right),D\left(p^{\mathrm{a}\left|\mathrm{g}\right.},c^{\mathrm{g}}\right)\right]$ onto $Q$.

end if

end for

end if

end if

end while

Create a RAG $G$ to depict the spatial relationship of all candidate superpixels ${\left\{{\Re }_i\right\}}_{i=1}^M$.

Refine the label map $L$ by Algorithm 3.

Return $L$.

Furthermore, there are several predefined operational details to ensure stability and efficiency. As shown in Algorithm 2, during the process of centroid splitting in a content-dense cluster, the newly generated centroids might fall on an already labeled pixel (i.e., located in other cluster regions). In this case, only the centroids located in regions that are identified as high content density by Equation (9) can be considered as new cluster seeds.

Algorithm 2 splitting operation within CATS

Input: Current cluster centroid $c_k$, current label map $L$

Output: Set of new cluster centroids $\Omega$.

Compute ${\left\{c_k^i\right\}}_{i=1}^4$ by Equation (11).

for each candidate seed $c_k^i$ in ${\left\{c_k^i\right\}}_{i=1}^4$ do

if $L\left(c_k^i\right)=0$ then

Set $\Omega =\Omega \cup \left\{c_k^i\right\}$.

end if

end for

Return $\Omega$.

In the merging process of two content-sparse clusters in Algorithm 3, if the area of the merged superpixel exceeds twice that of the average, it will no longer split. This is because the merging occurs after the splitting, and the newly generated centroid must fall on a labeled pixel.

Algorithm 3 merging operation within CATS

Input: Current Label map $L$ of candidate superpixels with the corresponding RAG $G$.

Output: Refined label map $L^{\prime }$.

while the number of nodes in $G$ is greater than $K$ do

Compute the area ratio $\theta$ corresponding to the global minimum weight ${\omega }_{mn}$.

if $\theta \ge 4$ then

for each pixel $p_i$ with label $L\left(p_i\right)=L\left(c_n\right)$ do

Set $L\left(p_i\right)=L\left(c_m\right)$.

end for

else

Set ${\omega }_{mn}=\infty$.

end if

Update $G$.

end while

Return $L^{\prime }$.

4. Experiments and Analysis

In this work, all experiments are implemented on an Intel i7-12700 processor with 2.1 GHz. This conducts comparative experiments on the Berkeley Segmentation Data Set 500 (BSDS500) [40], which is currently the most widely used dataset for superpixel generation. BSDS500 consists of 500 natural images with the size of $481\times 321$ or $321\times 481$, including animals, buildings, people, and other objects. Each image in the dataset contains at least five human-annotated ground truths, which enable convenient and accurate quantitative evaluation of superpixel performance. Meanwhile, MSLIC and SNIC are utilized as baselines, as well as USEQ, BASS, IBIS and DBSCAN are introduced as up-to-date competitors. To ensure fairness, all algorithms are configured with default parameters recommended by the original works.

4.1. Visual Comparison

The qualitative performance is visually compared in this part. Figure 4 shows four images and the corresponding ground-truths in the BSDS500 dataset, and Figure 5 exhibits the segmentation results of seven superpixel algorithms, wherein white outlines are boundaries of different labels. Obviously, all algorithms could produce superpixels with visually appealing shapes except for BASS, which generates irregularly shaped superpixels with uneven sizes. Furthermore, USEQ superpixels are more sensitive to textured content, resulting in unstable boundaries with sinuous and disordered shapes. On the other hand, most algorithms fail to accurately capture boundaries of small objects and strip areas (such as the bird and twig). For example, both MSLIC and IBIS superpixels could nearly adhere to the boundary of the bird through several iterations, but the accuracy is still low. As for other iteration-free methods, SNIC and DBSCAN, they also assume a uniform content complexity on the image plane, which neglect the local difference of information distribution. In contrast, the proposed CATS eases the constraints of the local optimum in the non-iterative clustering though online updating of seed placement (amount and position). Despite adopting a SNIC-like framework, CATS achieves better boundary fitting and uniform shape representation in these areas.

It is also worth noting that several algorithms exhibit limited accuracy on fitting boundaries within regions containing intricate structures or low contrast information (like the worm and sun). Owing to the adequate resampled seeds, small distinctions within these regions could be depicted by the CATS algorithm. Consequently, it demonstrated superior boundary adherence and exhibited heightened sensitivity towards areas encompassing complex structures, resulting in regular-shaped generated superpixels.

Apart from the aforementioned performance, another outstanding advantage of CATS is the segmentation under a limited amount of superpixels. Figure 6 demonstrates the consistency between the ground truth and the outlines of CATS superpixels for more intuitive comparison of boundary precision. It is obvious that almost all boundary pixels of image objects can be detected by superpixels with moderate over-segmentation. Specifically, CATS superpixels effectively preserve intricate structures in accordance with the actual outlines of different objects. This is done by reallocating the grid-level sampled seeds, which can both consider global image property and local content awareness. Therefore, it successfully reconciles the inherent contradiction between retaining accurate details and maintaining an optimal number of superpixels simultaneously.

4.2. Metrical Evaluation

As mentioned in the introduction, a desirable superpixel generation method should effectively mitigate the property incompatibility while achieving synergistic optimization. To quantitatively evaluate the results, four popular metrics are introduced from the extended Berkeley segmentation benchmark [41], including Boundary Recall (BR), Under-Segmentation Error (USE), Achievable Segmentation Accuracy (ASA) and COMpactness (COM). Specifically, 200 images in the test subset are selected to calculate the curves in Figure 7, which represent the average results ranging from 50 to 500 superpixels. It should be noted that the source code of BASS lacks the controllability of superpixel number; as a substitute, the expected superpixel number is adjusted to approximate the range. In addition, the source code of DBSCAN cannot batch the dataset when the expected superpixel number is less than 150.

Formally, let $T={\left\{T_m\right\}}_{m=1}^M$ and $\Re ={\left\{{\Re }_k\right\}}_{k=1}^K$ be the ground truth and the generated superpixels of $I$, respectively; the abovementioned metrics can be described as follows.

BR is versatile in evaluating the object segmentation and boundary detection. Mathematically, it is the ratio of ground truth boundaries covered by superpixel outlines [42].

$$\mathrm{BR}\left(T,\Re \right)={\displaystyle \frac{{\displaystyle {\sum }_{p_i\in T_b}\Pi \left({\min }_{p_j\in {\Re }_b}{‖P\left(p_i\right)-P\left(p_j\right)‖}_2<r\right)}}{T_b}}$$

(14)

where $T_b$ and ${\Re }_b$ represent boundary pixels in $T$ and outline pixels in $\Re$, respectively. The indicator $\Pi \left(\cdot \right)$ returns the logic value whether the expression is true. The threshold $r$ is set to 2 in this paper, which means that an outline pixel can be identified as an object boundary if there is a ground truth boundary pixel near it with the distance of 2 by Equation (3).

USE indicates the overall degree of one object overlapped with a superpixel [43]. Different from BR, it utilizes segmentation regions instead of boundaries for evaluation.

$$\mathrm{USE}\left(T,\Re \right)={\displaystyle \frac{{\displaystyle \sum _{m=1}^M\left({\displaystyle {\sum }_{{\Re }_k|{\Re }_k\cap {\Re }_m\ne \varphi }\left|{\Re }_k\right|}\right)}-N}{N}}$$

(15)

As a result of the region over-segmentation, a superpixel region ${\Re }_k$ is theoretically able and only able to cover one object. Therefore, the smaller the USE, the fewer boundary pixels of objects that are mistakenly included in ${\Re }_k$.

ASA quantifies the achievable accuracy of subsequent steps such as image segmentation and target recognition [44]. It is the proportion of valid segmentation regions generated by superpixels in the image.

$$\mathrm{ASA}\left(T,\Re \right)={\displaystyle \frac{{\displaystyle \sum _{k=1}^K\arg {\max }_m\left|{\Re }_k\cap T_m\right|}}{{\displaystyle \sum _{m=1}^M\left|T_m\right|}}}$$

(16)

If a superpixel only intersects with one ground truth region, the entire superpixel is considered as a valid segmentation region; conversely, if a superpixel intersects with multiple ground truth regions, the largest intersection is taken as the valid segmentation region. Therefore, when using superpixels as the basic units for image segmentation instead of individual pixels, the ASA value represents the upper limit of achievable accuracy.

COM can effectively describe the regularity of superpixel shapes [45]. Based on the isoperimetric theorem, the value of superpixel compactness is defined as the weighted isoperimetric quotient.

$$\mathrm{CO}\left(\Re \right)={\displaystyle {\sum }_{{\Re }_k}\frac{\left|{\Re }_k\right|}{N}}\cdot Q\left({\Re }_k\right),$$

(17)

Within k-th superpixel ${\Re }_k$, let $\boxed{{\Re }_k}$ and $\left|\boxed{{\Re }_k}\right|$ denote the collection of boundary pixels and the corresponding perimeter. The isoperimetric quotient is calculated by

$$\mathrm{Q}\left({\Re }_k\right)={\displaystyle \frac{\left|{\Re }_k\right|}{\pi {\left(\frac{\left|\boxed{{\Re }_k}\right|}{2\pi }\right)}^2}},$$

(18)

The denominator of the right term is the area of a perfect circle with an equal perimeter, and its weight in global compactness represents the proportion of its own size to the overall image.

The BR curves are all depicted in Figure 7a. A higher BR metrical value indicates a closer approximation between the superpixel edge and the actual image boundaries. It is obvious that CATS performs on par with USEQ and DBSCAN, which are both top-rank in terms of BR. Meanwhile, with further increases in the number of superpixels, the performance may potentially improve. On the other hand, four iteration-demand algorithms exhibit relative lower performances, which is partly because of the incompatibility between segmentation accuracy and feature quality. The constraint of spatial compactness term might induce misclassification of unlabeled pixels during clustering, which is then accumulated gradually within the subsequent iterations.

Figure 7b displays how USE varies within each algorithm. In general, a lower value indicates a higher proportion of each superpixel intersecting with only a single object. It adopts region-level results to evaluate the segmentation accuracy. Apparently, CATS achieves the best performance among all algorithms. Owing to the splitting-based re-clustering, the under-segmented regions can be easily updated with new superpixels that avoid overlapping with multiple objects. USEQ acquires the best BR but performs backward on USE. Since BR evaluates the degree of ground truth boundaries detected by the superpixels that do not consider false detection, both USEQ and DBSCAN generate dense but not precise boundaries.

The difference of ASA is plotted in Figure 7c. A greater ASA value indicates a higher proportion of correctly segmented parts. The closer the segmentation result of an algorithm is to the ground truth, the better the segmentation effect. Compared with other algorithms, CATS established an early performance advantage even under a limited superpixel number. Intrinsically, it benefits from the advantages of online averaging clustering that is also utilized in SNIC; therefore, they are similar in the trend ASA curve. When the number increases, the variance of image content can be captured by CATS more easily; as a result, it gradually exceeds SNIC and becomes the top-most one.

Figure 7d compares the compactness among different algorithms as the number of superpixels increases. A higher COM value signifies greater compactness within each superpixel region. It can be observed that CATS lags behind MSLIC and BASS, whose performance on BR, USE and ASA are inferior among all algorithms. On the other side, by re-clustering the irregular superpixel candidates with complex image content, CATS optimizes the COM of SNIC to a certain extent. As shown in Figure 5, compared with the other two iteration-free methods (USEQ and DBSCAN), CATS superpixels are trimmer and clearer, providing a better visual quality. It is also worth noting that, as the most related work to CATS, SNIC makes a balanced and considerable trade-off among these metrics. Nevertheless, the grid-level seeding strategy ignores the uneven distribution of image information, resulting in the downside of clustering small objects with rich details. Conversely, CATS could overcome this inherent shortcoming through the centroid relocation strategy. Moreover, it shows stronger controllability on superpixel number (i.e., the production is approximate or identical to the expectation) since the merging process can dynamically adjust the amount of superpixel candidates.

Execution Time (ET) is measured as the average runtime of 200 selected images in the test subset of BSDS500.

Table 1. Comparison of execution times among different algorithms (milliseconds). The italic numbers represent the lowest values.

Method	Expected Superpixel Number
	50	100	150	200	250	300	350	400	450	500
MSLIC	166	171	178	180	180	178	186	185	189	190
BASS	190	213	235	244	260	272	279	289	295	299
IBIS	21	20	18	18	18	18	18	18	18	18
SNIC	33	33	33	33	34	34	34	34	34	34
USEQ	25	25	25	25	25	25	25	25	25	25
DBSCAN	-	-	-	31	31	30	30	29	29	29
CATS	40	37	37	37	37	37	37	36	37	38

illustrates the comparison of all algorithms on the same platform. For fair comparison, no multithread or parallel programming is utilized in each implementation (the multithreading in USEQ is manually disabled).

In theory, both MSLIC and BASS work in the conventional SLIC-like manner, whose time consumption comes into two aspects, the computation complexity in a single loop and the total iteration times. Despite baseline work on SLIC showing $O(N)$ complexity, the practical calculations of pixel-centroid correlation within the clustering process of MSLIC and BASS are more sophisticated than the linear Euclidean distance measurement. Accordingly, there is a heavy computation burden that lags an order of magnitude behind the running efficiency.

As an exception in the iteration-demand superpixel algorithm, IBIS benefits from the reduction of pixels participating in the calculation of distance from neighboring pixels within each iteration. Consequently, it becomes the fastest method that even outperforms three iteration-free approaches. On the other hand, as shown in Figure 5, the improvement of IBIS in efficiency comes at the cost of mediocre segmentation accuracy and feature quality. In fact, IBIS merely maintains an approximate quality level as SLIC, which is proved to be significantly inferior than SNIC.

It can be also observed that USEQ exhibits a stable and favorable ET in a wide range of expected superpixel numbers. Since all pixels are processed in USEQ through a one-pass pattern in the MAP estimation, its complexity is $O(N)$, which works strictly independent with the number of superpixels. Nevertheless, the extreme performance is not applicable to some high-precision requirements. Similar situations may also occur in DBSCAN, which is corroborated by Figure 5.

Notably, despite SNIC concatenating the process of assigning and updating the results in faster convergence than SLIC, it showcases inferior efficiency compared to USEQ and DBSCAN. This is partly because of the existence of redundant computations of correlation measurements on pixels between two adjacent superpixel regions. To maintain the simplicity, CATS follows the conventional online averaging clustering framework in SNIC even if it can be accelerated through several effective strategies [43]. As listed in Table 1, it costs an additional 10% runtime to further adjust the SNIC superpixel candidates, which is still 4 times faster than the other baseline MSLIC on average.

In summary, the proposed CATS superpixels exhibit desirable outperformance compared with other up-to-date algorithms, particularly in terms of under-segmentation error and compactness. To acquire content awareness, CATS performs fine-grained segmentation on regions with detailed context while merging smaller partitions with low region complexity. Consequently, as the number of superpixels increases, both interior homogeneity and boundary adherence improve and consistently remain at the forefront. Unlike the conventional grid-level seeding strategy, CATS superpixels achieve a desired number of superpixels without parameter adjustment. Additionally, it outperforms SNIC, IBIS, and DBSCAN and USEQ in terms of spatial compactness. Therefore, it can be employed as an advantageous pre-processing tool for various tasks, offering a comprehensive performance between feature performance and running efficiency.

5. Conclusions

This work proposes an optimized superpixel algorithm termed CATS to generate compact and regular superpixels with content adaptation. By leveraging the property of area elements in manifold feature space, a novel centroid splitting and merging operator is devised in CATS. A majority of seeds are then resampled based on image content density during the updating process. Accordingly, more seeds are allocated in regions with higher content density to generate smaller superpixels, while relatively fewer seeds are assigned in areas with lower content density to generate larger superpixels. In addition, with the advantage of label expansion patterns in the online averaging clustering framework, redundant pixel inspecting and correlation measurement operations can be effectively avoided. Experimental results demonstrate that the proposed CATS achieves structure sensitivity with visual satisfaction, and delivers superior segmentation accuracy compared to other SOTA algorithms.

Future research will primarily focus on exploring strategies to further enhance the efficiency of CATS and apply it to several advanced computer vision tasks.