Partition and Inclusion Hierarchies of Images: A Comprehensive Survey

Abstract

The theory of hierarchical image representations has been well-established in Mathematical Morphology, and provides a suitable framework to handle images through objects or regions taking into account their scale. Such approaches have increased in popularity and been favourably compared to treating individual image elements in various domains and applications. This survey paper presents the development of hierarchical image representations over the last 20 years using the framework of component trees. We introduce two classes of component trees, partitioning and inclusion trees, and describe their general characteristics and differences. Examples of hierarchies for each of the classes are compared, with the resulting study aiming to serve as a guideline when choosing a hierarchical image representation for any application and image domain.

1. Introduction

Properties such as accuracy, size and relation between elements are considered for an image representation in accordance with the application domain. We list several such families of representations with their characteristics:

• Pixel-based representation of an image is characterized by a large number of elements in simple neighborhood relations [1] and contains uninterpreted intensity information [2].
• Block-based representations (on binary images [3,4,5] and grayscale images [6,7,8]) divide the image into the set of (rectangular) arrays of pixels. It has fewer elements than pixel-based representations, but the image data are still not interpreted. Most common applications include image compression [3,6,7], segmentation [5,6], and feature and attribute extraction [4,5,8].
• Compressed (or frequency) domain representations store the image as a set of coefficients in a transform domain, such as Fourier transform [9,10], wavelets [10,11,12], ridgelets [13], contourlets [14], etc. Typical uses include image compression [10,11], denoising [13,15], reconstruction [12], texture analysis and segmentation [16]. Their advantage is a reduced image size, but they are sensitive to translation, rotation and scaling [17] and make it difficult to manipulate localized image content.
• Region-based representations group similar connected pixels using a segmentation algorithm, typically producing an over-segmentation. The resulting regions are often called superpixels [18], with the region adjacency information kept in a region-adjacency graph (RAG) [19] or combinatorial maps [20]. Different approaches include normalized cuts [21], graph-based segmentation by Felzenszwalb and Huttenlocher [22] and watershed segmentation [23,24], with a comparison in [18]. The reduced number of regions based on interpreting image information keeps the representational accuracy [2], but further unions of regions need to be considered to detect semantic structures [25].
• Hierarchical representations propose those most likely unions of regions on different scales of the image, from fine to coarse [25]. They enrich the horizontal relations between regions present in (partial) segmentations [26,27] with vertical inclusion information between regions at different scales. Initially, they were used for image filtering [2,28,29], segmentation [30,31,32] and boundary detection [33] but have since been applied to a vast amount of domains and applications (cf. Table 3).

We focus on the hierarchical representations of image in this work, and refer to them as component trees due to their aim to propose complex regions corresponding to “meaningful” image objects [28,29]. While hierarchical image representations have been receiving increasing amounts of attention [34,35,36] in image processing community, the structure, properties and construction of such hierarchies are the focus of study of Mathematical Morphology. Quad-trees [32,33] were amongst the first developed hierarchical image representations. However, they are regular hierarchies with predefined image division which is not dependent on the image content, and are in the same relation to the hierarchies considered in this paper as the block-based to the region-based representations. Recently, many different such hierarchical representations have been developed: Trees of Shapes [37,38,39], Binary Partition Trees (BPT) [2,25] and trees based on them (e.g., BPT by Altitude Ordering [40], Hierarchies of Minimum Spanning Forests [41]), $\alpha$-trees [42,43] and constrained connectivity hierarchies, such as $(\omega )$-trees [42,44] which merit a structured study and comparison.

Contributions. The “building blocks” of hierarchical image representations were thoroughly investigated by Serra [26] and Ronse [27], who study the (partial) segmentations and partitions, and further the lattices of those (partial) partitions forming hierarchies [45,46,47]. In contrast to these and other theoretical works from Mathematical Morphology presented in a more general framework (e.g., [48]) and appropriated for different domains, we examine the component trees built directly from monochannel images represented by vertex-valued graphs and equipped with a 4-connectivity.

This paper formalizes the general characteristics of hierarchical representations, and introduces two distinct superclasses based on the common properties of the currently used representations (inclusion and partitioning trees, briefly introduced in [49]). While hierarchies provide the information relating to the image structure in terms of object relations, they need to be indexed in order to study the scale of image content. We present a consistent way of indexing the hierarchies based on dendrograms [50], and additionally propose a mapping from indexed inclusion hierarchies onto partitioning hierarchies on a doubled domain. Different hierarchies are compared in terms of structure, duality, parameterization, their ability to represent objects, and representation completeness.

Existing surveys covering several hierarchical informations are presented in the next section, followed by the basic notions used throughout the article in Section 3. Section 4 formalizes the hierarchies and proposes a unifying hierarchy classification and indexing, while Section 5 is dedicated to a detailed analysis of different hierarchies and their characteristics. A comparison between the current construction algorithms, including references, serving as user guidelines when making an implementation choice is given in Section 6. Finally, Section 7 offers a summary and comparison of different characteristics presented in Section 5 and Section 6, as well as identifies the open challenges based on the presented concepts, while the work will be concluded in Section 8.

2. Related Work

Despite the recent popularity of hierarchical image representations, the published works summarizing and comparing their characteristics tend to be limited to either a specific domain or hierarchy type. In contrast, we aim to explain the general properties of popular hierarchical representations and their behavior in different domains such as filtering, segmentation and object detection.

Hierarchies are often used to select a specific or optimal image (partial) partition for further processing [51]. Partitions and partial partitions are constituent parts of hierarchical representations, studied in the unifying papers by Serra [26] and Ronse [27] which present the theory of (partial) partitions and study the lattices they make and their relations. The hierarchies are studied and formally defined in [45] as an erosion from the lattice of levels to that of partial partitions. In contrast to our proposition to study the hierarchy structure and scale separately, the definition of a hierarchy offered by Ronse [45] refers directly to indexed hierarchy. We offer both definitions for each studied hierarchy in order to facilitate their use in the theoretical framework of Ronse [45], further expanded to study the ordering between partitions forming hierarchies [46,47].

Soille [42] examined multiple approaches to hierarchical image partitioning, focusing on the problems caused by the chaining effect in the $\alpha$-hierarchy and proposing constrained connectivity solutions. The studied hierarchies are compared in the context of image simplification, comparing the number of regions based on simplification strength (level) as well as visually comparing the segmentation results. Although the paper proposes many potential modifications to $\alpha$-connectivity, presently, only the global range constraints are widely used, forming a hierarchy known as $(\omega )$-tree.

An exhaustive survey of filtering tools called connected operators was offered by Salembier and Wilkinson [52], focusing on their characteristics and techniques used to create them, and encompassing both the “classical” filtering approaches based on pixel-based representations as well as the approaches based on component trees. Approaches based on simplifying component trees with pruning and non-pruning strategies act on the connected components where the image is constant, bridging the gap between filtering and segmentation. Several hierarchies are presented in the strong context of filtering. It deals with extrema-oriented filtering based on inclusion trees, complemented only by a Binary Partition Tree for regions of intermediate level and without focusing on the implications of replacing an inclusion hierarchy with a partitioning one.

In contrast, the paper by Najman and Soille [44] formalizes the structure of hierarchical partitioning representations in the context of image segmentations, extending on the definitions of hierarchical clustering while taking spatial relations between the image elements into account. The ultrametric property of hierarchical clustering is explained in the context of hierarchical segmentations, emphasizing the importance of object scale. Hierarchies are represented by dendrograms [50], and the ultrametric watershed is introduced as a new visualization and computation scheme for partitioning hierarchies. We extend the framework to inclusion hierarchies, but regard the structural information of a component tree separately from the scale information in an indexed hierarchy.

One of most recent surveys, by Cousty et al. [40], is presented in the framework of edge-weighted graphs (i.e., operating on the dissimilarities between image pixels, as opposed to working directly with pixel values). It is accompanied by a paper which presents efficient quasi-linear algorithms based on the Kruskal Minimum Spanning Tree algorithm for computing these hierarchies [53], and establishes the relations between the $\alpha$-tree of an image with other Mathematical Morphology hierarchies and segmentation from markers. Recently, these links were studied further in [54] and put additionally in relation with saliency maps, images which can be interpreted as characteristic maps of hierarchies used for visualization, accompanied by example applications both from the field of image analysis and outside of it.

An application independent survey paper was presented by Najman and Cousty [55] covering a larger scope of morphological image processing techniques. It presents the basic concepts in modern Mathematical Morphology, offering introductory discussions on several topics. They discuss in detail different connectivities which can be used to define the image graph as well as the basic morphological operators. Image segmentation and connected filtering techniques leading to hierarchies of partitions are also presented through examples linking the $\alpha$-tree hierarchy, minimum spanning forests and watersheds on edge-valued graphs, offering an intuitive insight into various image processing tasks performed on hierarchies. Contrary to this work, which presents the reader with different choices of connectivity relations and filtering criteria but only provides a general idea of the capabilities of hierarchical image processing, our work uses a straightforward framework of 4-connectivity in order to better emphasize the differences and properties of hierarchies under examination.

The published surveys are mostly application-dependent, deal with a limited range of hierarchical representations and do not contrast well the hierarchies of partitions and partial partitions. The goal of this paper will thus be to provide an application independent reference about the structure and characteristics of commonly used hierarchical representations, decoupling the structure of the hierarchy from the scale of the represented regions. It is aimed at a general image processing audience, as a guide in comparing a large range of different hierarchical representations and aid when making a choice of representation.

3. Image Representation

This section describes the primary definitions from graph theory following [56], explains the interpretation of images as graphs and finally defines the terminology used in Mathematical Morphology to refer to image components. Firstly, in Section 3.1, we revise how images, partitions and regions are represented on a graph. Secondly, in Section 3.2, we define further terminology for handling image regions, as well as some characteristic image regions.

3.1. Images as Graphs

A graph comprises points and arrows which join one point to another. More formally, a graph $\mathcal{G}$ is an ordered pair $(V,U)$, where V denotes a set of points, called vertices (or nodes) and U a family of arrows between those points (i.e., duplicate elements are allowed), called arcs. Two vertices joined by an arc are said to be adjacent. An edge is obtained from an arc by discarding the direction of said arc. If the directions of arcs between the vertices are not specified, it is more convenient to represent the graph as a pair $\mathcal{G}=(V,E)$, where E stands for the set of edges, and such a graph is then referred to as an undirected graph. An undirected graph is a simple graph if it has no loops (edges from a vertex to itself) and has no more than one edge between any two vertices. While all results applicable to undirected and directed graphs can be applied to both types of graphs indiscriminately, for convenience, it is easier to distinguish between the two types depending on the application. Further, let I be a grayscale digital image which comprises a set of pixels, and $f_I:I\to {\mathbb{N}}_0$, where $N_0=\{0,1,2,\dots \}$ is the set of natural numbers including 0. A function that assigns to each pixel $p\in I$ its intensity value. The dual image, denoted by $-I$, comprises the same set of pixels while the intensity function is changed: $f_{-I}(p)=\mathit{lMax}-f_I(p)$, where $\mathit{lMax}$ is the maximum allowed gray level.

When representing an image I as a graph, the image pixels correspond to the set of vertices V, and choosing an adjacency relation or connectivity (cf. [1] for more details on connectivity and [57,58,59,60] for examples of advanced connectivities) determines the set of edges E connecting those pixels and associates an undirected graph $\mathcal{G}=(V,E)$ to the image. An edge between two adjacent pixels p and q is denoted by $e_{p,q}$ or $e_{q,p}$. When using regular 4-connectivity, we can distinguish the image boundary pixels as those pixels $p\in I$ that do not have the full set of 4 neighbors. For a theoretical analysis of connectivity and connections, the theory of connective segmentation [26] and its extension to partial connections [27] permit combining different connections.

A path $\mathcal{P}$ in $\mathcal{G}$ from $p_1$ to $p_n$ is defined by a sequence of vertices (pixels) $(p_1,\dots ,p_n)$ such that for all $1\le i<n$ the pixels $p_i$ and $p_{i+1}$ are adjacent. A path from $p_1$ to $p_n$ is a cycle if $p_1=p_n$ and has no duplicate edges. $\mathrm{SP}(p,q)$ denotes the set of all possible paths in $\mathcal{G}$ between any p and q. Pixels p and q are connected in $\mathcal{G}$ if and only if there is a path $\mathcal{P}$ in $\mathcal{G}$ from p to q or if $p=q$.

We next define the concepts involved into image segmentation based tasks. A subgraph of $\mathcal{G}$ generated by a set of vertices $V_{\mathcal{X}}\subseteq V$ is defined as $\mathcal{X}=(V_{\mathcal{X}},E_{\mathcal{X}})$, where $E_{\mathcal{X}}=\{e_{p,q}|p,q\in V,e_{p,q}\in E\}$. A partial graph of $\mathcal{G}$ is generated by a set of edges $E_{\mathcal{X}}$ and defined as $\mathcal{X}=(V,E_{\mathcal{X}})$ (i.e., all the vertices and a selected set of edges from the original graph.) Finally, we denote a partial subgraph of $\mathcal{G}$ by $\mathcal{X}\subseteq \mathcal{G}$, and define it as a subgraph of a partial graph, $\mathcal{X}=(V_{\mathcal{X}},E_{\mathcal{X}})$. Such $\mathcal{X}\subseteq \mathcal{G}$ is said to be connected if all $p,q\in V_{\mathcal{X}}$ are connected in $\mathcal{X}$.

A region $\mathcal{R}=(V_{\mathcal{R}},E_{\mathcal{R}})$ of I generated by a nonempty $\mathcal{X}\subseteq \mathcal{G}$ is defined by $V_{\mathcal{R}}=V_{\mathcal{X}}$, $E_{\mathcal{R}}=\{e_{p,q}|e_{p,q}\in \mathcal{G}\wedge p,q \mathrm{are} \mathrm{connected} \mathrm{in} \mathcal{X}\}$. A connected region or a connected component of the image I is a region $\mathcal{R}\subseteq \mathcal{G}$ which is connected and of maximal extent, i.e., $\nexists {\mathcal{R}}^{\prime }\subseteq \mathcal{G}\mathrm{s}.\mathrm{t}.\mathcal{R}\subset {\mathcal{R}}^{\prime }$. For a region $\mathcal{R}$, one can similarly define connected components as connected regions ${\mathcal{R}}^{\prime }\subset \mathcal{R}$ of maximal extent. We denote the set of all connected components of the image as $\mathrm{CC}(I)$, or a region $\mathcal{R}$ as $\mathrm{CC}(\mathcal{R})$. Unless explicitly specified, all the partial subgraphs and regions in the remainder of the article will be connected.

A set of regions $\pi =\{{\mathcal{R}}_1,\dots ,{\mathcal{R}}_k\},k\ge 1$ is said to partition the image if it covers the entire image domain and the elements of the set are mutually disjoint:

$$\begin{array}{cc}\hfill \left(\bigcup _{{\mathcal{R}}_i\in \pi }{V}_{{\mathcal{R}}_i}\right)& =V=I\hfill \\ \hfill \mathrm{and}& \\ \hfill \forall {\mathcal{R}}_i,{\mathcal{R}}_j\in \pi ,i\ne j,& V_{{\mathcal{R}}_i}\cap V_{{\mathcal{R}}_j}=\varnothing \hfill \end{array}$$

(1)

It can be also defined through an equivalence relation or by associating each point to a class [26,61]. The intensity function $f(·)$ is partitioned using one of many existing image segmentation algorithms [26]. The regions need not be connected (i.e., segmentation to foreground and background, into predetermined classes [62,63,64], second generation connectivity [57,65]). However, in case of simple 4-connectivity, one would typically work with connected components of the regions for the purpose of building the component trees from the regions.

A segmentation can also determine boundaries between the regions, or contain a residual. If the region set returned by an algorithm does then not cover the entire image domain and is not a partition, it can be handled in the framework of partial partitions introduced by Ronse [27]. Similar to Equation (1), the elements of the set ${\pi }^{*}=\{{\mathcal{R}}_1,\dots ,{\mathcal{R}}_k\},k\ge 1$ partially partition the image if they are disjoint, but need not cover the image domain. In fact, the set:

$$\mathbf{supp}({\pi }^{*})=(\bigcup _{{\mathcal{R}}_i\in {\pi }^{*}}{V}_{{\mathcal{R}}_i})$$

(2)

is called the support of the partial partition, and ${\pi }^{*}$ partitions the image on its support $\mathbf{supp}({\pi }^{*})$. Both the family of partitions and family of partial partitions of a set form a complete lattice, with the ordering between partitions given by:

$${\pi }_1\le {\pi }_2\iff \forall {\mathcal{R}}_i\in {\pi }_1,\exists {\mathcal{R}}_j\in {\pi }_2,{\mathcal{R}}_i\subseteq {\mathcal{R}}_j.$$

(3)

This relation defines the refinement order on partitions, and an equivalent relation defined between any two partial partitions ${\pi }_1^{*}$ and ${\pi }_2^{*}$ gives the standard order. The greatest element of the lattices is the partition into one region containing the whole set, while the smallest element is the empty partition for the lattice of partial partitions, and a partition into singleton elements for the lattice of partitions. The lattices are studied in detail in [26,27].

3.2. Manipulating Image Components

A region boundary of a region $\mathcal{R}=(V_{\mathcal{R}},E_{\mathcal{R}})$ is defined as the set of edges $E_{\mathrm{bound}}(\mathcal{R})$ given by:

$$E_{\mathrm{bound}}(\mathcal{R})=\{e_{p,q}\in E|p\in V_{\mathcal{R}},q\notin V_{\mathcal{R}}\}.$$

(4)

The sets of pixels of the inner and outer boundary are then made out of all the end-points of the boundary edges that belong (resp. do not belong) to the region $\mathcal{R}$:

$$V_{\mathrm{inbound}}(\mathcal{R})=\{p\in V_R|e_{p,q}\in E_{\mathrm{bound}}(\mathcal{R})\}$$

(5)

$$V_{\mathrm{outbound}}(\mathcal{R})=\{p\in V|p\notin V_{\mathcal{R}},e_{p,q}\in E_{\mathrm{bound}}(\mathcal{R})\}.$$

(6)

All the image pixels not belonging to a connected region $\mathcal{R}$ ($p\in V$ but $p\notin V_{\mathcal{R}}$) make a set of 0 or more connected regions of maximal size in the image domain I, $\overline{\mathcal{R}}=\{\overline{{\mathcal{R}}_1},\dots ,\overline{{\mathcal{R}}_k}\},k\ge 0$. If a $\overline{{\mathcal{R}}_i}$ does not contain any image boundary pixels, it is called a hole of the region $\mathcal{R}$ (cavity in case of 3D images). The operation of filling all the holes of a connected region, $H(·)$, adds all the pixels contained in all the holes of a region $\mathcal{R}$ to that region $\mathcal{R}$:

$$\begin{array}{cc}\hfill H(\mathcal{R})=\mathcal{R}\bigcup \left(\bigcup _i\overline{{\mathcal{R}}_i}\right),& i\ge 0\hfill \\ \hfill \mathrm{such}\mathrm{that}& \forall i,\overline{{\mathcal{R}}_i}\mathrm{is}\mathrm{a}\mathrm{hole}\mathrm{in}\mathcal{R}.\hfill \end{array}$$

(7)

Flat zones of the image are connected components of the image I on the partial subgraph of $\mathcal{G}=(V,E)$ generated by the set of edges $E_0$ such that $e_{p,q}\in E_0\iff f(p)=f(q)\wedge e_{p,q}\in E$, that is connected regions of maximal size comprised only of pixels at the same gray level [66,67]. Flat zone $F_k$ at a gray level k can be described as:

$$\begin{array}{cc}\hfill F_k=\{p_1,\dots ,p_l\},& l\ge 1\hfill \\ \hfill \mathrm{such}\mathrm{that}& \forall p_i\in F_k,f(p_i)=k\hfill \\ \hfill \mathrm{and}& \forall p_j\notin F_k,\mathrm{if}{e}_{p_i,p_j}\in E \mathrm{then}f(p_j)\ne k.\hfill \end{array}$$

(8)

We call a flat zone of the image a local maximum $F_k$ if it is surrounded by pixels of strictly lower gray level:

$$\forall p\in V,\left(e_{p,q}\in E,q\in F_k,p\notin F_k\right)\Rightarrow f(p)<k.$$

(9)

Local minima are flat zones surrounded by pixels of strictly higher gray level. Local minima and local maxima together make the local extrema of the image.

The upper and lower level sets ($L^k$ and $L_k$, respectively) of an image are sets of image pixels with gray level values higher or lower than k:

$$L^k=\{p\in I|f(p)\ge k\}$$

(10)

$$L_k=\{p\in I|f(p)\le k\}$$

(11)

We additionally define ${\mathcal{F}}_k,{\mathcal{L}}^k$, and ${\mathcal{L}}_k$ as subgraphs generated by $F_k,L^k,L_k$, respectively, while still referring to them as flat zones or level sets of the image, and note that they each comprise several connected components.

4. Component Trees

This section provides a framework for the definition and indexing of hierarchies as well as highlights the construction differences between two types of hierarchies. First, Section 4.1 recalls the definitions of hierarchies on partitions and partial partitions following the formalization in [45,46]. Then, Section 4.2 highlights practical construction differences and region evolution within typical hierarchies, classifying them into partitioning and inclusion hierarchies. Finally, we describe the framework for indexing such hierarchies in Section 4.3 according to the specificities of the two classes.

4.1. Hierarchies of Partitions and Partial Partitions

Partitions and partial partitions form lattices, defined by the refinement order for partitions, or the standard order (containing the merging, inflating and inclusion orders) for partial partitions (cf. [46] for a detailed study on these orders). Thus, in morphological image segmentation a hierarchy is typically defined as an increasing sequence of (partial) partitions going from the least to the greatest element of the lattice: ${\pi }_0\le \dots \le {\pi }_n=1_E$, where the least element is ${\pi }_0=0_E$ (partition to singletons) for partitions and ${\pi }_0=⌀$ (empty partition) for partial partitions [46,68]. In practice, all the non-singleton regions at the hierarchy bottom can be treated as singleton elements.

However, the definition through a sequence of (partial) partitions does not decouple the shape and inclusion information from the scale inposed on the regions by indexing (i.e., assigning levels to) the hierarchy. We define a hierarchy without indexing it, similarly to [69]:

• $\mathcal{G}\in \mathcal{H}$,
• for each two elements ${\mathcal{R}}_1,{\mathcal{R}}_2\in \mathcal{H}$ the following holds: ${\mathcal{R}}_1\cap {\mathcal{R}}_2\ne \varnothing \Rightarrow {\mathcal{R}}_1\subset {\mathcal{R}}_2$ or ${\mathcal{R}}_2\subset {\mathcal{R}}_1$.

Elementary regions do not need to be included, which allows the support of the partial partition to grow through the hierarchy. This definition imposes the partial ordering on the regions only, contrary to the one through a sequence of (partial) partitions imposing an order on the partitions building the hierarchy. In fact, any set of regions for which these rules hold defines a hierarchy, as we are only interested in the unique hierarchy components. We further justify it by the fact that in case of selecting optimal image segmentation from a hierarchy, one needs to consider the regions of the hierarchy separately rather than the (partial) partitions building it (studied in [2,70,71] with the most recent unifying theory by Kiran and Serra [51,72]). It corresponds to selecting optimal cuts in the hierarchies based either on externally imposed criteria (i.e., energy imposed to partitions) [51,70] or ground truth images [72,73].

To model the structure of a hierarchy, we deal with what berge [56] called arborescences or rooted trees, which are defined on directed graphs. In directed graphs, the arcs are ordered pairs of elements and an arc $a_{n,m}$ from n to m does not imply the existence of $a_{m,n}$. A quasi-strongly connected graph $\mathcal{G}$ is such a graph where for every pair of vertices $p,q$ there exists a vertice r such that there is a path from r to p and from r to q, and a rooted tree $\mathcal{T}=(M,P)$ is a quasi-strongly connected acyclic graph. The definition of a path $\mathcal{P}$ in a tree $\mathcal{T}$ and a graph $\mathcal{G}$ is similar, except that the arcs have to be directed $a_{p_i,p_{i+1}}$ for all consecutive elements $p_i,p_{i+1}$. If there exists an arc $a_{m,n}$ in a tree, m is called the parent of n and n a child of m. The vertex m is called an ancestor of n if there exists a path $\mathcal{P}$ in $\mathcal{T}$ from m to n. The only node which has no children in $\mathcal{T}$ is called the root of the tree, while the nodes that have no children are called leaf nodes. Let $C=\{n_1,\dots ,n_k\}$ be a set of nodes. C is a cut of the tree if every path $\mathcal{P}$ from the root to any leaf passes through exactly one node $n_j\in C$.

In a tree ${\mathcal{T}}_{\mathcal{H}}=(M,P)$ which is a component tree, the nodes of the tree model the regions of the hierarchy and the arcs model their inclusion. The region represented by a node $n\in M$ is denoted by $\mathcal{R}(n)=(V(n),E(n))$. The root node $r\in M$ corresponds to the whole image, while leaf nodes $n\in M$ are such nodes for which $\nexists m\in M\mathrm{s}.\mathrm{t}.V(m)\subset V(n)$. A hierarchy shown in Figure 1a is modeled by the component tree in Figure 1b.

The relation between the parent region and the regions represented by its children can be formalized as follows. If m is a parent node in the tree and $n_1,\dots ,n_k$ are all the children of m, the following rules describe how to construct m from its children:

$$V(m)=V(n_1)\cup \dots \cup V(n_k)\cup S(m),$$

(12)

where

$$\begin{array}{cc}\hfill S(m)=\{p_0,\dots ,p_l\},& l\ge 0\hfill \\ \hfill \mathrm{such}\mathrm{that}& \forall i\in \{0,\dots ,l\},p_i\in I\hfill \\ & \forall j\in \{1,\dots ,k\},p_i\notin V(n_j).\hfill \end{array}$$

(13)

The edge set of the parent can be represented as:

$$E(m)=\{e_{p,q}\in E|p\in V(m),q\in V(m)\},$$

(14)

and the pixel set $S(m)$ has to be such that the following holds:

$$\mathcal{R}(m)=(V(m),E(m))\mathrm{is} \mathrm{a} \mathrm{connected} \mathrm{region} \mathrm{of} I.$$

(15)

Equation (12) states that a pixel set of the parent can be written as a union of the pixel sets of all its children, and optionally a set of additional pixels $S(m)$. Equation (15) ensures the connectedness of the resulting union, and Equation (14) that $\mathcal{R}(m)$ is indeed a region with the vertex set $V(m)$. This describes the rules one needs to follow when constructing tree regions in a bottom-up approach.

To reconstruct an image from the associated component tree, an altitude or model needs to be assigned to each node n which introduces new pixels into the hierarchy, i.e., which is a leaf, or for which $S(n)\ne \varnothing$. When forming an image, gray level is assigned to all pixels as they are introduced to the hierarchy, at the altitude of the corresponding node. Scale information is imposed by indexing the hierarchy, discussed in Section 4.3. an indexed hierarchy corresponds to the definition through a sequence of (partial) partitions.

We also note that an extension to the framework of hierarchies of (partial) partitions has been proposed by Kiran and Serra [74] through introducing braids of partitions. A braid is a family of partitions larger than a hierarchy, such that the pairwise supremum of any two partitions of the braid $\mathcal{B}$ forms a cut in some monitoring hierarchy $\mathcal{H}$. A single braid can posses multiple monitoring hierarchies, while any hierarchy is a braid with itself as a monitor. By definition, dynamic programming properties valid on hierarchies are still valid on braids of partitions. Braids of partitions are a tool allowing to combine and fuse multiple hierarchies calculated on different modes of a multimodal image. This structure however lies outside of the framework of partitions and partial partitions [46] (and in fact extends it), as it is more than a hierarchy. As such, further discussion exceeds the scope of this paper and the interested reader is referred to [74,75] for the theory of braids of partitions, and to [76] for their application to multimodal image segmentation. We also direct the reader interested in combining different hierarchies of the same image into a new hierarchy (rather than a braid or some other structure) to [54] where such a method was proposed for the hierarchies of partitions.

4.2. Construction Differences between Inclusion and Partitioning Hierarchies

Equations (12)–(15) are general relations formalizing the hierarchy through a component tree. While the hierarchies are naturally divided into hierarchies of partitions and those of partial partitions, in current literature the hierarchies of partial partitions tend to be further restricted. We propose the names of partitioning trees for the hierarchies on the lattice of partitions, and inclusion trees for the subset of the hierarchies on the lattice of partial partitions, based on the nature of parent–child relations in the representations. Such a categorization was previously proposed in [49] and is explored here in more detail with added illustrations and explanations Any hybrid superclasses would necessarily be hierarchies of partial partitions. an example would be hierarchies of quasi-flat-zones or $\alpha$-connected components constrained by minimal area threshold (the authors thank one of the reviewers for this example.) However, to the best of our knowledge, all the currently studied and used hierarchies can be assigned into one of the two proposed superclasses. This difference between the classes is shown in Figure 2.

Inclusion trees. The leaf nodes of inclusion trees hold small regions or points, such as local image maxima or minima [28,29,37], or markers on the image [77]. In a typical bottom-up construction, nodes are formed by a region growing process starting from the leaves by adding one or more pixels (in Mathematical Morphology, this commonly means the image flat zones) to the existing regions. The new node becomes a parent of the nodes merged by this action and this continues until the root node is obtained covering the whole image domain. As the regions are only formed by adding new pixels to existing region(s), we constrain Equation (13) to a strict inequality, $l>0$.

Basic image simplification or filtering [52] using an inclusion tree includes cutting branches from the leaves up to a region satisfying a desired criterion and collapsing all the removed regions into their closest surviving ancestor. Inclusion trees in literature are based on gray level or contrast between regions, and as such model dark and bright image structures. In this case, the filtering removes small contrasted structures in the image without changing the larger structures.

Partitioning trees. The principal difference with inclusion trees is that the leaves, as well as any cut, of a partitioning tree forms an image partition [70]. The initial partition can be any oversegmentation of the image such as image pixels, flat zones [25,42] or watershed segmentation [78]. New regions of the inner nodes are formed as unions of two or more existing adjacent regions. To reflect this, a constraint $k>1$ is added to Equation (12) and $l=0$ to Equation (13).

At the core of the partitioning tree construction are the notions defining iterative merging [2], introduced by Garrido et al. [30]:

• Region model defines how regions and their unions are represented. It reflects the characteristics of the regions used in the construction process.
• Merging criterion or similarity (or dissimilarity) measure describes the interest of possible merges. It is based on the region model.
• Merging order defines the rules used to merge the regions and which merge to perform next based on the merging criterion.

Simplifying or filtering an image using a corresponding partitioning tree corresponds to selecting a cut in the tree based on some criterion and assigning a model (or altitude) to each node in the selected cut to be used in image reconstruction. As partitioning trees are not typically extrema-oriented and can represent regions of intermediate gray levels, filtering the tree removes the small variations in the gray levels of regions perceived as uniform.

4.3. Indexing the Hierarchy

A component tree defines any hierarchy through its structure. To fully exploit the scale and semantic information provided by the different construction algorithms, the hierarchy is indexed by assigning an attribute $\lambda$, called the level (of aggregation), to each node of the component tree. It is always a non-negative function of the nodes, increasing along each branch from the leaves towards the root, meaning that if m is an ancestor of n, then $\lambda (n)<\lambda (m)$. An indexed hierarchy corresponds to the definition [46] through an increasing sequence of (partial) partitions. It can be equivalently defined as an erosion from the lattice of levels to the lattice of (partial) partitions, with an adjoint dilation assigning a level to each region of the hierarchy. This definition is given alongside the definition through an indexed component tree for each presented hierarchy in Section 5 for comparison.

A distance on a set of elements $\Omega$ is a function satisfying the conditions of non-negativity, $d(v_1,v_2)\ge 0$, identity, $d(v_1,v_1)=0$, symetry, $d(v_1,v_2)=d(v_2,v_1)$ and triangular inequality, $d(v_1,v_2)\le d(v_1,v_3)+d(v_3,v_2)$ for any elements $v_1,v_2,v_3\in \Omega$ of the set. A distance is an ultrametric distance if it obeys an ultratriangular inequality, a constraint stronger than the triangular inequality: for any three elements of a set, $v_1,v_2,v_3\in \Omega$, it is true that $d(v_1,v_2)\le \max (d(v_1,v_3),d(v_2,v_3))$. A bijection between indexed partitioning trees and ultrametric distances [69,79,80] on the same set is established for the case where $\lambda (n)=0$ if and only if n is a leaf node. This is in accordance with the construction algorithms of different partitioning trees, which tend to assign the same attribute value (usually $\lambda =0$) to all the leaf nodes. The levels induce an ultrametric distance on all the vertices of an image graph $\mathcal{G}=(V,E)$ with a corresponding partitioning tree $\mathcal{T}$, which can be expressed as:

$$d(v_1,v_2)=\min \{\lambda (n)|n\in \mathcal{T},v_1\in V(n),v_2\in V(n)\}$$

(16)

A distance between any two image elements from I is given by the smallest level of a node n in the hierarchy representing a region containing both image elements. Such indexed trees are represented in a form of a dendrogram [50] (introduced under the name taxonomic tree [81] for hierarchical clustering), with node height corresponding to the level assigned to it (cf. Figure 3).

For inclusion trees and hierarchies of partial partitions in general the identity and non-negativity constraints become $d(v_1,v_2)\ge d(v_1,v_1)\ge 0$. The dendrogram of an inclusion tree is more complicated as points can enter the support of the partial partition at levels above 0 [46], which needs to be indicated even when they are entering the support by the process of growing (inflating) an existing region. An example of such dendrogram is shown in Figure 4c.

Adding the corresponding sets $S(n)$ as nodes at the lowest level of the hierarchy will allow us to construct a partitioning tree from an inclusion tree, however we will be unable to distinguish the original from the added singletons or reconstruct the original inclusion hierarchy. This is similar to the problem encountered in [27] when trying to represent an output of a segmentation algorithm producing a residual in the framework of connective segmentations and partitions: adding the residual to the representation by supplementing the partial partition with singletons does produce a full partition, but makes the residual indistinguishable from the original partition components, promoting the introduction of the framework of partial partitions [27]. To avoid the loss of information, we propose to uniquely map an inclusion tree onto a partitioning tree with a double image domain. We extend every pixel with a ghost pixel at the same gray level, connected only to the original pixel in the image. All the pixels and their ghost pixels form the leaves of the tree and partition the new domain. A ghost pixel merges into the hierarchy at the same level (in the same node) as its original pixel does, ensuring that every node in the tree is formed by merging two or more nodes. The singletons in the hierarchy correspond to the auxilary nodes present due to the ghost regions, making it possible to distinguish between them and the nodes of the original hierarchy. We can now impose an ultrametric distance on the nodes of this new hierarchy if we assign all the auxiliary nodes the level 0. To ensure $\lambda (n)>0$ for all other nodes, we add an arbitrary constant to the level assigned by the construction algorithm, and then Equation (16) also holds for indexed inclusion hierarchies. This mapping is of general interest as it allows applying techniques developed for partitioning trees with inclusion trees, e.g., the technique of [82] on combining several hierarchies of the same image, and in general processing the inclusion trees in the more common framework of multiscale segmentations.

5. Analysis of Tree Characteristics

This section reviews the tree representations frequently used in current literature, offering their definitions as well as examining, summarizing and comparing their characteristics. First, the trees from the inclusion tree superclass are listed, followed by partitioning trees. The most efficient algorithms suited for implementation will be mentioned in Section 6.

5.1. Min and Max-Trees

We first explain the concept and give examples for the Min-tree, followed by explaining the implications of the duality relation with the Max-tree. The Min-tree is an inclusion tree representing the dark structures in the image, based on lower level sets with the leaves corresponding to local image minima. A connected component of the level set ${\mathcal{L}}_k$ not present in the level set ${\mathcal{L}}_{k-1}$ makes a new node $n_k$ with a region $\mathcal{R}(n_k)=(V(n_k),E(n_k))$, which either becomes:

• a parent node to all the previously constructed nodes at lower levels which are included in the region of the new node: $\mathcal{R}(n_{k^{\prime }})\subset \mathcal{R}(n_k)$, $k^{\prime }<k$; or
• a leaf node if it does not include the regions of any previous nodes.

At the highest gray level, there is only one connected component covering the whole domain and forming the root of the tree. The hierarchy corresponds to distinct connected components of the lower level sets: ${\mathcal{H}}_{\mathit{min}}=\{\mathrm{CC}({\mathcal{L}}_k)|k\le \mathit{lMax}\}$, shown in Figure 5b. The local minima of the dual image $-I$ correspond to the local maxima of the original. Thus, considering the upper level sets of I or the lower level sets of the dual $-I$ we obtain the dual Max-tree hierarchy, defined as ${\mathcal{H}}_{\mathit{max}}=\{\mathrm{CC}({\mathcal{L}}^k)|k\le \mathit{lMax}\}$. The Min-tree is suitable for manipulation of dark image structures, and the Max-tree of the bright ones. However, keeping both representations to handle both types of object simultaneously as the trees are redundant and ensuring consistency between them is difficult [83].

The connected components of ${\mathcal{L}}_k$ form a partial partition of the image, $\mathrm{CC}({\mathcal{L}}_k)$, thus the map $k\mapsto \mathrm{CC}({\mathcal{L}}_k)$ is an erosion from the lattice of levels to the lattice of partial partitions on the image defining the Min-tree. It assigns the level k to the connected component of ${\mathcal{L}}_k$ forming a node in the hierarchy. The level k is also used as the altitude, carrying the information about the darkest region pixel. To index the Max-tree, the erosion and levels are defined using the dual image $-I$ instead, example of which is shown in Figure 5d,e. The altitude of a node $n_k$ at level k is calculated as $\mathit{lMax}-k$.

The structure was introduced by Salembier et al. [29] as a Max-tree and by Jones [28] as a component tree, which only differ in how the representation is stored [84]. More complex hierarchies based on these trees were developed to use with advanced connectivity relations, such as the Dual-Input Max-Tree for mask-based connectivity [57,65] and hierarchies for hyperconnectivity [59]. Min-trees have also been calculated on inclusion hierarchies instead of images, to perform filtering operations in shape spaces [85].

5.2. Tree of Shapes

The Tree of Shapes (ToS) is an inclusion tree combining the Min and Max-tree [37,86] to represent both bright and dark structures simultaneously. The leaves correspond to all the local extrema, while the node regions hold shapes, obtained by applying the hole filling operator $H(·)$ on all connected components of the lower and upper level sets. We can thus define the ToS hierarchy as ${\mathcal{H}}_{\mathit{ToS}}=\{H(\mathcal{R})|\mathcal{R}\in {\mathcal{H}}_{\mathit{min}}\}\cup \{H(\mathcal{R})|\mathcal{R}\in {\mathcal{H}}_{\mathit{max}}\}$. These shapes are either nested or disjoint [38,87]. A node n is the parent of node m if there is no shape ${\mathcal{R}}^{\prime }$ such that $\mathcal{R}(m)\subset {\mathcal{R}}^{\prime }\subset \mathcal{R}(n)$. For the original image in Figure 5a, the 5 distinct shapes obtained from upper and lower sets in Figure 6a,b is shown in Figure 6c, with the final hierarchy in Figure 6d. The ToS is a self-dual structure, as the sets of local minima and maxima as well as the upper and lower level sets are inverted in the dual image $-I$, which then produces the same ToS. The redundancy is eliminated by using the modified connected components (i.e., shapes) when combining the trees.

To index the Tree of Shapes and obtain the dendrogram in Figure 6e, we rely on the notion of dynamics along a path (introduced as a contrast measure [88]) and the notion of region dynamics. The dynamics along the path $\mathrm{PD}(·)$ is defined as the sum of intensity differences of adjacent pixels along the path:

$$\mathrm{PD}(\mathcal{P})=\sum _{i=2}^n|f(p_i)-f(p_{i-1})|$$

(17)

Region dynamics $\mathrm{RD}(·)$ of a region $\mathcal{R}(n)$ is chosen between all minimal dynamics along the paths between pixels in the same branch of the tree, as the largest such path dynamic:

$$\mathrm{RD}(\mathcal{R}(n))=\max \{\min \{\mathrm{PD}(\mathcal{P})|\mathcal{P}\in \mathrm{SP}(p,q)\}|p\in S(n),q\in \mathcal{R}(n)\}.$$

(18)

It can also be defined recursively:

$$\begin{array}{cc}\hfill \mathrm{RD}(\mathcal{R}(n))& =\max \{\left|p-q\right|+\mathrm{RD}(\mathcal{R}(m))|\forall m \mathrm{child} \mathrm{of} n,p\in S(n),q\in S(m)\}\hfill \\ \hfill \mathrm{RD}(\mathcal{R}(n))& =0\mathrm{if} n \mathrm{is} \mathrm{a} \mathrm{leaf} \mathrm{node},\hfill \end{array}$$

(19)

which allows for easy level calculation in a bottom-up tree construction. The level of the node n associated to the region $\mathcal{R}(n)$ is then equal to the region dynamics of the region. To define the hierarchy as an erosion, we define the dynamic shapes at level k:

$$D_k=\{p\in V|\mathcal{R}=(V,E)\in {\mathcal{H}}_{ToS},\mathrm{RD}(\mathcal{R})\le k\}.$$

(20)

We also define ${\mathcal{D}}_k$ as the subgraph generated by $D_k$, and obtain the map $k\mapsto \mathrm{CC}({\mathcal{D}}_k)$ as the erosion from the lattice of levels to the lattice of partial partitions indexing the ToS hierarchy. The ToS is a complete image representation, where the node altitudes correspond to the level h of the region from the upper or lower level set (resp. ${\mathcal{L}}^h$ or ${\mathcal{L}}_h$) which formed the shape $\mathcal{R}(n)$, assigned to the node n. The altitude can be calculated from the levels if the gray levels of the leaves and the sign of the region dynamic for the inner nodes in Equation (19) are stored.

This type of structure was independently presented by Monasse and Guichard [86] for a combination of 4- and 8-connectivity, and by Song and Zhang [38] for 6-connectivity (proven equivalent in [89]). A reduced version of this hierarchy was also proposed by Song and Zhang [38,89], aiming to represent only the changes in the topology of the image. The name “Tree of Shapes”, used here, was chosen because it prevails in recent literature [39,83].

5.3. Binary Partition Tree

Binary Partition Tree (BPT) is a partitioning tree and is not extrema oriented [2], making it suited for representing objects with low, high and intermediate gray levels. The construction starts with assigning a node to each region from the initial partition which is based on pixels, flat zones or precomputed partitions [2,30]. The hierarchy depends on the choice of region model and merging criterion. The region model can be as simple as the average or median gray level of a region, and is typically assumed to be constant within a single region. A merging criterion is defined as a dissimilarity measure between the region models. The merging order used for the BPT is that of iterative region merging, and dictates the merging of the two regions most similar according to the merging criterion in each step (the order is not defined in the case of more than one pair of most similar regions). The merging sequence and the resulting tree for a BPT using a simple region model and criterion are shown in Figure 7.

The tree is not usually well balanced (cf. [2,25]) and can contain simple regions very close to the tree root. Thus, without any assumptions to the similarity measure used in merging, we propose an indexing in such a manner that the similarity measure of each merging step can always be recovered:

• if a node m of $\mathcal{T}$ is a leaf node, then $\lambda (m)=0$,
• if a node m is created as a union of regions corresponding to the nodes $n_1$ and $n_2$ (i.e., $\mathcal{R}(m)=\mathcal{R}(n_1)\cup \mathcal{R}(n_2)$), and the dissimilarity between the regions in the moment of merging was $D(\mathcal{R}(n_1),\mathcal{R}(n_2))$, the level of the new node m is calculated according to:

  $$\lambda (m)=\max (\lambda (n_1),\lambda (n_2))+D(\mathcal{R}(n_1),\mathcal{R}(n_2)).$$

  (21)

A corresponding dendrogram for the tree in Figure 7e is shown in Figure 7f. The most precise reconstruction of the original image equals to the precision provided by the initial partition. While the self-duality of the BPT depends on the used region model and merging criterion (if based on gray or color level), it is typically desirable (the merging criterion used in Figure 7 is one such example). As the BPT is the most generic out of all the presented trees and does not pose any restrictions to the merging criterion, we only define the indexing of the nodes recursively due to the complexity of the closed-form solution as an erosion from the lattice of levels to the lattice of partitions.

The BPT is the most versatile hierarchy presented in this article, indroduced in Mathematical Morphology by Salembier and Garrido [2] by keeping track of the merging steps of iterative merging [30]. This work was continued in [25] by proposing a more suitable merging criterion and applying it to object detection, while the BPT and appropriate simplification methods were studied in [78]. Many developed extensions [41,53,90,91] highlight the relations to other component trees and morphological techniques, and specialize it according to the application domain. Additionally, many hierarchical representations outside of the morphological community fit well into the BPT framework, using segmentation methods such as watershed variants and Felzenszwalb-Huttenlocher segmentation to determine the initial partition, and defining the region models and similarity measures based on region boundary, color, texture, size and depth information [34,35,36]. We mention here the two notable specializations of the BPT, which have been used to produce hierarchies over images represented by edge-weighted rather than vertex-weighted graphs considered in this work.

Binary Partitioning Trees by Altitude Ordering establish the relation between min-trees (cf. Section 5.1) calculated on the edge graph of the image and $\alpha$-trees (cf. Section 5.4). They are an intermediate step in the calculation of the $\alpha$-tree (or quasi-flat-zone) hierarchy and other hierarchies when the edge values are calculated as the intensity differences between neighboring pixels, including additionally an imposed order between the edges of equal value, typically defined through Kruskal’s Minimum Spanning Tree algorithm [92]. Such a hierarchy was first explored by [93] where different segmentation methods based on the principal hierarchy were also proposed, as well as a possibility of different valuations of the graph edges. The concept was reintroduced to the community and formalized under the name of BPT by Altitude Ordering by Najman et al. [53] and Cousty et al. [54] where the relations to other hierarchies were established.

The Hierarchies of Minimum Spanning Forests establish the links with watersheds and segmentation from markers. They are also primarily defined on an edge-weighted graph of the image, but additionally require a set of markers ordered by importance. The markers are typically chosen as local minima of the weight map, when the MSF hierarchies correspond to watershed segmentation defined through the drop of water principle [24]. If the minima are additionally ordered by their extinction values [29,94] based on attributes such as area, depth or volume, the resulting Hierarchies of MSF are called hierarchical watersheds [41,54,77]. Hierarchies of MSF allow reordering of the hierarchy according to the calculated or perceived importance of minima, and have as such been used as 2D and 3D segmentation tools as well as for processing of geographical data [54].

5.4. $\alpha$-Tree

The second presented partitioning tree is the $\alpha$-tree. Unlike for the BPT, which relies on a predefined merging order but can be customized by the choice of initial partition, region model and merging criterion, all of these parameters are strictly defined for the $\alpha$-tree. We examine here the $\alpha$-tree for gray level images in detail, while the proposed adaptations for multichannel images will be discussed briefly in Section 7.2, along with other open challenges.

The initial partition in the $\alpha$-tree is always the partition to image flat zones. Using the notions introduced in Section 4.2 to describe the $\alpha$-tree, the region model, merging criterion and merging order are defined as follows:

• The region model for each region $\mathcal{R}$ is the boundary of that region, $E_{\mathrm{bound}}(\mathcal{R})$.
• The merging criterion defines the similarity between two neighboring regions as the lowest edge (valued by gray level difference) common to models of both regions: $D(\mathcal{R},{\mathcal{R}}^{\prime })=\min \{|f(p)-f(q)\left|\right|e_{p,q}\in E_{\mathrm{bound}}(\mathcal{R}),e_{p,q}\in E_{\mathrm{bound}}({\mathcal{R}}^{\prime })\}$.
• The merging order dictates that, in the i-th step, all regions with the similarity equal to i should be merged.

We also give a more natural definition through $\alpha$-connected components (also called quasi-flat zones). Image flat zones are considered as 0-connected components. To define the $\alpha$-connected components for some $\alpha >0$, the connected components are considered on the image graph where the neighboring pixels are connected if their gray level difference is less than or equal to $\alpha$. For increasing values of $\alpha$, these regions form a hierarchy represented by a tree. If we denote an $\alpha$-connected component to which a pixel p belongs to by $\alpha \text{-CC}(p)$, the hierarchical relation between the regions can be expressed as:

$$\alpha \text{-CC}(p)\subseteq {\alpha }^{\prime }\text{-CC}(p)\forall \alpha \le {\alpha }^{\prime }$$

(22)

The $\alpha$-connected components for an image in Figure 8a are shown in Figure 8b–e, with the $\alpha$-tree shown in Figure 8f. To index the $\alpha$-tree (cf. example in Figure 8g) we define the quasi-flat-zone partition at level t as ${\mathrm{QZFP}}_t=\bigcup \{\alpha \text{-CC}(p)|\forall p\in I,\alpha \le t\}$, and the map $t\mapsto {\mathrm{QZFP}}_t$ as the corresponding erosion from the lattice of levels to the lattice of partitions.

The locality of the similarity metric which considers only the neighboring pixels can cause gray level variations within a single region to be higher than expected for a certain $\alpha$. An example is shown in Figure 8f, where the level $\alpha =2$ contains both a region with 18 and 2 pixels. This is referred to as the chaining effect and the extreme case is shown in Figure 9. Different approaches have been proposed to address this issue [42,44,95,96,97].

This tree is self-dual as the contrast information between neighboring pixels does not change in the dual image $-I$, and a complete image representation. Simplifying the images by approximating the regions in a partition formed by respecting a local range parameter was introduced in [98], studied further in [99], while the term $\alpha$-connected component was first used by Soille in [42,100] where the hierarchical properties of such regions were asserted.

5.5. $(\omega )$-Tree

For the ease of understanding, we first provide the usual definitions of the $(\omega )$-tree through the $\alpha$-connected components of maximal size, followed by the more complex definition through the region model, merging criterion and merging order. The $(\omega )$-tree is a partitioning tree aiming to solve the problems with chaining effect present with $\alpha$-trees. If the parameter $\alpha$ restricts the maximal range between locally connected pixels, $\omega$ corresponds to the maximal global gray level range inside a connected component. The global gray level range of a component is denoted by $\mathrm{GR}(·)$ and is defined as the value difference between the highest and lowest pixel gray level value in the component. The notion of $(\alpha ,\omega )$-connected components was introduced together with the notion of $\alpha$-connected components by Soille [100], denoted by $(\alpha ,\omega )\text{-CC}(p)$ for a pixel p, and defined as the ${\alpha }^{\prime }\text{-CC}(p)$ with the maximal possible ${\alpha }^{\prime }\le \alpha$ such that the global range is still lower or equal to $\omega$:

$$\begin{array}{cc}\hfill (\alpha ,\omega )\text{-CC}(p)& ={\alpha }^{\prime }\text{-CC}(p),\hfill \\ \hfill \mathrm{where}{\alpha }^{\prime }& =\max \{{\alpha }^{″}|{\alpha }^{″}\le \alpha \mathrm{and}\mathrm{GR}({\alpha }^{″}\text{-CC}(p))\le \omega \}.\hfill \end{array}$$

(23)

Even though the following relation holds for every pixel p:

$$(\alpha ,\omega )\text{-CC}(p)\subseteq ({\alpha }^{\prime },{\omega }^{\prime })\text{-CC}(p)\mathrm{for}\alpha \le {\alpha }^{\prime }\mathrm{and}\omega \le {\omega }^{\prime },$$

(24)

the set of all $(\alpha ,\omega )$-connected components can not form a hierarchy due to the unknown order between $(\alpha ,\omega )\text{-CC}(p)$ and $({\alpha }^{\prime },{\omega }^{\prime })\text{-CC}(p)$ for $\alpha \ge {\alpha }^{\prime }$ and $\omega \le {\omega }^{\prime }$.

For $\alpha \ge \omega$, the local range parameter $\alpha$ does not play a role, as the $(\alpha ,\omega )$-connected components are equivalent to those for $\alpha =\omega$ [42]. The $(\omega )\text{-CC}$ are considered the largest $\alpha \text{-CC}(p)$ with the global range less or equal to $\omega$:

$$(\omega )\text{-CC}(p)=(\alpha \ge \omega ,\omega )\text{-CC}(p)=\max \{{\alpha }^{\prime }\text{-CC}(p)|\mathrm{GR}({\alpha }^{\prime }\text{-CC}(p))\le \omega \}.$$

(25)

Unlike the components defined by Equation (23), the $(\omega )\text{-CC}$ define a hierarchy, called the $(\omega )$-tree. an example of this hierarchy for the image in Figure 8a is depicted in Figure 10.

To define the $(\omega )$-tree through region model, region criterion and merging order, there are the following changes as when compared to the $\alpha$-tree:

• The region model for $\mathcal{R}$ is the boundary of that region, $E_{\mathrm{bound}}(\mathcal{R})$, and minimal and maximal pixel values: $\min \{|f(p)-f(q)\left|\right|e_{p,q}\in \mathcal{R}\},\max \{|f(p)-f(q)\left|\right|e_{p,q}\in \mathcal{R}\}$.
• The merging criterion defines the similarity between all allowed sets of regions $\mathcal{N}=\{{\mathcal{R}}_1,\dots ,{\mathcal{R}}_n\},n\le 2$ such that the subgraph of I generated by the union of all the region vertice sets: ${\bigcup }_{i=1}^n{V}_{{\mathcal{R}}_i}$ is connected. The weight of such a merge between a set of regions $\mathcal{N}$ is:

  $$\begin{array}{cc}\hfill W(\mathcal{N})=& \max (\min \{|f(p)-f(q)||\hfill \\ & e_{p,q}\in E_{\mathrm{bound}}({\mathcal{R}}_i),e_{p,q}\in E_{\mathrm{bound}}({\mathcal{R}}_j),\forall i,j,i\ne j\},\hfill \\ & \max \{f(p)|p\in {\mathcal{R}}_i\forall i\}-\min \{f(p)|p\in {\mathcal{R}}_i\forall i\}).\hfill \end{array}$$

  (26)
• The merging order dictates that in the i-th step, all the sets of regions of maximal extent with the similarity equal to i should be merged.

Indexing the ($\omega$)-tree is similar to indexing the $\alpha$-tree since they are both constrained connectivity hierarchies [42]: we define global range partitions as ${\mathrm{GRP}}_t=\bigcup \{(\omega )\text{-CC}(p)|\forall p\in I,\omega \le t\}$ and $t\mapsto {\mathrm{GRP}}_t$ as the corresponding erosion (cf. Figure 10g for dendrogram). The problems with the chaining effect are not completely solved since the $(\omega )$-decomposition of the image can not contain regions not already present in the $\alpha$-decomposition, however the $(\omega )$-indexing better reflects the region complexity (compare $\omega =8$ with $\alpha =1$ for the full image in Figure 9c).

Since it is based on the $\alpha$-tree, the $(\omega )$-tree retains its self-dual property and other characteristics. The $(\omega )$-tree of the same image always has more levels, and the number of regions decreases more slowly from the leaf levels towards the root. However, as both trees represent only the unique regions, the $(\omega )$-tree representation always has fewer nodes than the $\alpha$-tree since regions are removed from the $\alpha$-tree when forming an $(\omega )$-tree. Both of these effects can be observed by comparing the dendrograms from Figure 8g and Figure 10g.

6. Construction Algorithms

We discuss here the approaches to construct various trees, explain the main idea behind each algorithm and provide the references to the original approaches. The complexity is expressed in relation to the number of image pixels N, depending on quantization defined by the number of bits used to code pixel values q and the range of those values $k=2^q$.

Min- and Max-tree construction. a recent comparison of Min-tree and Max-tree construction algorithms was offered by Carlinet and Géraud [101], dividing the construction algorithms into immersion algorithms, flooding algorithms and merge-based algorithms. The merge-based approaches are mainly used for parallelism, and are not further discussed here (for a recent parallel implementation of Max-tree combining the merge-based and flooding approach, cf. e.g., [102]).

The first efficient Min-tree computation algorithm by Salembier et al. [29] is a flooding algorithm. It begins with the pixel at the gray level $\mathit{lMax}$ which constitutes a root pixel, from which the depth-first propagation through image elements is performed, recursively [29] or using a hierarchical queue [103]. For low quantization ($q\le 16\mathrm{bit}$) a linear complexity of $\mathrm{O}(kN)\sim \mathrm{O}(N)$ is achieved since k is small, but for larger q, the complexity becomes quadratic, $\mathrm{O}(N^2)$, making them unsuitable for generic pixel (e.g., float) values. To overcome this, Wilkinson [65] replaces the hierarchical queue with a priority-queue, resulting in a flooding algorithm running in $\mathrm{O}(N\log N)$ complexity for any pixel type with a total order (e.g., high dynamic range integers and float values).

The first step of immersion algorithms is sorting all the N image elements according to the appropriate order (i.e., the intensity ordering), which then form N disjoint singleton sets. Those sets are then merged to form a tree in the second step, using Tarjan’s union-find algorithm [104]. The complexity of the sorting step is bounded to $\mathrm{O}(N+k)$ for small integers (typically $q\le 12\mathrm{bits}$ or $q\le 16\mathrm{bits}$), but increases to $\mathrm{O}(N\log N)$ for generic data types. For low quantized data, i.e., $q\le 12\mathrm{bit}$, the union-find implementations with lowest complexity by Berger et al. [84] and Najman and Couprie [105] use techniques such as root path compression, union-by-rank and level compression (to the detriment of memory usage) to achieve a quasi-linear time complexity of $\mathrm{O}(N\times \alpha (N))$ ($\alpha (N)$—very slow growing inverse of the diagonal Ackermann’s function, $\alpha ({10}^{80})\simeq 4$). For arbitrarily quantized data, the sorting step binds the complexity to $\mathrm{O}(N\log N)$.

Tree of Shapes construction. Early approaches to ToS construction operated with worst-case time complexity of $\mathrm{O}(N^2)$ [83,89] and were not easily extendible to multidimensional ($nD$) images [37]. A recent algorithm by Géraud et al. [39] overcomes these drawbacks by using the immersion algorithms for Min-tree construction as a canvas, and replacing the sorting step.

The sorting step in an immersion algorithm should place the image elements from the external shapes before the elements from the internal shapes. For the ToS computation, representing the image as a set-valued map on a Kahlimsky grid [106] materializes the inter-pixel spaces and enables the computation of the correct shape order. The sorting step depends on a hierarchical queue, so can only handle quantized data, and runs in $\mathrm{O}(kN)$ considered linear for low quantized data (authors recommend $q\le 12\mathrm{bits}$). The immersion step has the same $\mathrm{O}(N\times \alpha (N))$ complexity as for the Min-tree, resulting in the quasi-linear complexity of $\mathrm{O}(N\times \alpha (N)+kN)\sim \mathrm{O}(kN)$ for the whole algorithm.

Binary Partition Tree construction. The BPT computation starts with an initial partition, then merges the two most similar regions in each step (until reaching a single region), iteratively updating the similarity. We present here the complexity estimation of the construction algorithm for the case when updating the region model and calculating similarity between the models can be done in constant time, as studied by Guigues [107].

When no specific connectivity is imposed on the image, the iterative construction algorithm corresponds to hierarchical (agglomerative) clustering [108]. The detailed analysis of the algorithm identifies the advantages of using a hierarchical queue to keep the similarity information [30]. The initial partition is a segmentation into N separate pixels or regions. The similarities of the initial over-segmentation are sorted after calculation, which is bounded by $\mathrm{O}(N^2\log (N^2))\sim \mathrm{O}(N^2\log N)$ for up to to $N^2$ similarities. The construction proceeds by iteratively finding the pair of most similar regions, removing the similarity information related to those regions, merging them and re-calculating the similarity information for the new region. Determining the smallest similarity is a constant operation in a hierarchical queue, while the similarities are updated in $\mathrm{O}(N\log N)$, with the number of iterations equal to the number of regions. Based on this, the construction complexity of BPT is bounded by $\mathrm{O}(N^2+N^2\log N+N\times (N+N\log N))\sim \mathrm{O}(N^2\log N)$. In the case of a specific connectivity, the speed of calculating and updating the similarity information can significantly decrease depending on the merging criterion used (4-connectivity analyzed in [109] approximates an upper bound to $\mathrm{O}(\frac{2}{3}{N}^2)\sim \mathrm{O}(N^2)$). The special cases of BPT based on the $\alpha$-tree [40,53] can be constructed even faster, relying on their properties achieved by specializing the BPT parameters.

$\mathit{\alpha }$-tree construction. The $\mathit{\alpha }$-tree construction algorithm relies on its equivalence with a Min-tree defined on the edges valued with pixel intensity differences [44,69], and can use any Min-tree algorithm. Extending the idea, Havel et al. [110,111] calculate the $\mathit{\alpha }$-tree directly using a modification of Tarjan’s union-find [104], presenting an algorithm suited for multithreading applications.

The algorithm by Najman et al. [53], inspired by Kruskal’s Minimum Spanning Tree algorithm [92] and using Tarjan’s union-find [104], first constructs a BPT by Altitude Ordering, from which the $\mathit{\alpha }$-tree can be obtained with a linear post-processing step [40,53]. The complexity of the $\mathit{\alpha }$-tree construction algorithms is the same as for the Min-trees, and is quasi-linear in the number of image pixels for low quantization, $\mathbf{O}(\mathit{N}\times \mathit{\alpha }(\mathit{N}))$.

($\mathbf{\omega }$)-tree construction. The first step in calculating the $(\mathit{\omega })$-tree and other constrained connectivity hierarchies is calculating the $\mathit{\alpha }$-tree they are based on. The ultrametric watersheds by Najman [69] enable visualizing these hierarchies and calculating them from the $\mathit{\alpha }$-tree based on the Lowest Common Ancestor (LCA) [112] in constant time for every pair of neighboring elements in the ultrametric watershed. The transformation into the $(\mathit{\omega })$-tree is itself linear.

The $(\mathit{\omega })$-tree can be interpreted as an $\mathit{\alpha }$-tree where the global component range is used as a region complexity measure according to Bosilj et al. [49], which proposes a filtering approach removing the nodes from the $\mathit{\alpha }$-tree and indexing the hierarchy by assigning the complexity measure as the level (of aggregation) to the nodes. This is realized by a bottom-up traversal of the tree, linear in the number of nodes of the tree and the number of image elements. Using any of the proposed approaches, the $(\mathit{\omega })$-tree can be constructed in the same complexity as the $\mathit{\alpha }$-tree, $\mathbf{O}(\mathit{N}\times \mathit{\alpha }(\mathit{N}))$ for low quantized image data.

7. Applications and Future Directions

This survey offers a comparison of component trees which is inclusive, not application-dependent and in the widespread image processing framework on vertex-valued graphs equipped with 4-connectivity. After the structure and the basic idea behind the construction algorithms of the trees were presented, this section focuses on current applications, interesting research directions and application domains for component trees. Section 7.1 offers an overview and comparison of the presented trees in terms of image processing applications, while the open challenges are outlined in Section 7.2.

7.1. Comparative Summary

For each tree, the characteristics such as duality, completeness and types of regions held in the representation were determined. The most basic choice is the one between inclusion trees and partitioning trees. While both types of hierarchies are well suited for image simplification, filtering and object detection, the structure of inclusion trees resulted in their common usage when selecting prominent image regions (such as discriminating keypoints) while the partitioning trees lend themselves better to producing segmentations of the image according to given criteria. As all the presented trees are complete representations, they can be either used directly as a search space, or their filtered versions or reduced hierarchies can be used if a smaller search space is needed.

The region types of different trees are closely examined as content representation has a direct influence on the type of objects and scenes each representation is well suited for. When working with extremal regions of the image, one should choose an inclusion tree representation. A choice between the different inclusion trees will depend on whether the contrast of the object related to the background is known in advance (for processing bright or dark objects, one would choose a Max-tree or a Min-tree respectively, but if this information is not known the ToS can be used to process both types of objects at the expense of slightly higher construction times). On the other hand, partitioning trees can contain regions at intermediate gray levels and are better suited for processing complex scenes (e.g., a satellite image of a whole city) and identifying composite objects. The most flexibility is offered by the BPT and its many specializations, where the construction parameters can be determined with a specific application in mind.

The construction complexity is another important factor to consider, and was examined for different quantization and data types in Section 6. All these characteristics are summarized for inclusion trees in

Table 1. Summary of characteristics for inclusion trees.

Tree	Max Tree	Min Tree	Tree of Shapes
Dual tree	Min tree	Max tree	self-dual
Type of objects	dark objects	bright objects	shapes
Complete representation?	Yes	Yes	Yes
Construction complexity	$\mathbf{O}(\mathit{N}\times \mathit{\alpha }(\mathit{N}))$	$\mathbf{O}(\mathit{N}\times \mathit{\alpha }(\mathit{N}))$	$\mathbf{O}(\mathit{kN})$
Additionalparameters	No	No	No

and partitioning trees in

Table 2. Summary of characteristics for partitioning trees.

Tree	Binary Partition Tree	$\mathit{\alpha }$-Tree	$(\mathit{\omega })$-Tree
Dual tree	self-dual ${}^{\mathit{a}}$	self-dual	self-dual
Type of objects	unions of initial partition	$\mathit{\alpha }$-CC (quasi flat zones)	$(\mathbf{\omega })$-CC
Complete representation?	Yes ${}^{\mathit{b}}$	Yes	Yes
Construction complexity	$\mathbf{O}({\mathit{N}}^2\mathbf{log}\mathit{N})$	$\mathbf{O}(\mathit{N}\times \mathit{\alpha }(\mathit{N}))$	$\mathbf{O}(\mathit{N}\times \mathit{\alpha }(\mathit{N}))$
Additional parameters	Initial partition,	No	No
	region model,
	similarity measure

${}^{\mathit{a}}$ Depends on region model and similarity measure, but usually desirable. ${}^{\mathit{b}}$ depends on the initial partition.

.

To provide a practical example of using these trees in image processing, a result of simple level-based filtering is shown in Figure 11. It shows the way different trees interact with image components, with the extrema-oriented inclusion trees processing the regions according to inclusion and gray-level order, and the partitioning hierarchies working on similarity-based regions with intermediate gray levels. On the one hand, a list of various application domains given in

Table 3. Some applications of the presented trees from the literature.

Application Domain	Inclusion		Partitioning
	Min/Max	ToS	BPT	$\mathit{\alpha }$-Tree	$(\mathit{\omega })$-Tree
filtering/simplification	[28,52,85,105,113,114,115]	[37,65,116,117,118]	[52]	[42,95]	[42,119]
image/video segmentation	[28,29,114,120]	[117,118,121,122,123,124]	[2,109,125,126,127,128]	[42,44,129,130]	[42,44,119]
image compression	[131]	[124,132]
object detection & tracking	[133,134]		[25,135,136,137]
feature/edge detection	[103]	[83,138,139,140]
pixel/feature description	[62,141,142,143,144]	[63,144,145]		[146]	[146]
image comparison		[37]
change detection	[147]
image retrieval	[103,142,148,149]	[140]			[150]
classification	[64,151,152]			[153,154,155]	[156]
time series processing			[157]
astronomical imaging	[84,114,134,158]
remote sensing (inc. hyperspectral imaging)	[62,64,141,143,144,147,149,152,159,160]	[63,144,145,155]	[126,127,128,135,136,161]	[153]	[119,156]

shows that simplification, description of whole images (for remote and hyperspectral imaging) and, naturally, segmentation, are prevalent applications with partitioning trees. On the other hand, dealing with isolated image features (and applying them to image retrieval), as well as filtering out small details and noise, or segmenting and detecting specific objects on a contrasted background (e.g., in astronomical imaging) was mostly done using inclusion trees.

7.2. Open Challenges

Hereafter, we identify some open challenges not discussed herein, which would provide viable directions for future research.

This article focuses mainly on monochannel images, but as the examined hierarchical structures have a primary goal of providing good estimated locations of objects contained in the image, the color (and any other kind of) information contained in multichannel images has to be considered. For a general partitioning tree, working with vectorial instead of scalar image element values is fairly easy, as the total order between the values of image elements is not required. As such, the BPT was originally defined with color in mind [2], and the trend of including color information has persisted in current literature [25,78]. For the $\mathit{\alpha }$-tree, several adaptations for multichannel images exist [42,95,119,129,162] (https://github.com/pbosilj/trees-lib).

However, for the inclusion trees, due to a wide range of available total orderings on image element values [163] for multivariate data, this extension is not straightforward. Max-trees and Min-trees for multivariate data were first investigated for general applications [158]. The study of relations between these hierarchies and multivariate data in the context of connected filtering conducted by Passat and Naegel [164] and Kurtz et al. [159] define a more general structure called a component graph for multivalued images, as well as specific constraints under which a component graph preserves the tree structure and is instead referred to as multivalued component tree. An algorithm for extracting distinguished features from color images [165] also indirectly defines a Max-tree and Min-tree structure for color images. Extension of the Tree of Shapes to color images has been only very recently tackled by Carlinet and Géraud [166]. Let us note that when the tree is used solely for image description, processing multichannel data (e.g., hyperspectral images) can be simply achieved with a marginal strategy, i.e., building a tree for each image channel (see [141] for Max-trees and Min-trees, [63] for the Tree of Shapes), or by an hybrid marginal/vectorial strategy [152].

We also note advances in the application domain of image filtering where the Max-tree hierarchy was used to implement connected operators [29]. A whole variety of connected operators can be realized by replacing the Max-tree with other component trees, and this fundamental extension to the framework of connected operators was studied in [52]. Recently, Xu and co-authors [85,139] have proposed defining the filtering in shape space, an image whose adjacency relation is given by the parental relations in its original Max-tree. A novel way of filtering and node selection is also proposed by Perret and Collet [114], which puts Mathematical Morphology in relation with stochastic models, relying on a Markovian algorithm to classify the nodes of a tree.

Other open problems include finding the solution to the chaining effect present in $\mathit{\alpha }$-trees (cf. Subsec Section 5.4). This first led to defining logical predicate connectivity [100] and constrained connectivity [42,44,95,96,97,153] and establishing the $(\mathit{\omega })$-tree (explained in Section 5.5). More recently, mask-based connectivity, hyperconnectivity and directed connectivity, as well as the advanced hierarchies based on them [59,65,167], were introduced as a way to handle both visually disconnected objects, overlapping or weakly connected objects in images.

An interesting challenge would also be to formalize the theoretical relations between the hierarchies. They are only mentioned here as implied by the construction algorithms in Section 6. Examples include building the $\mathit{\alpha }$-trees as the Min-tree of the edges [69,110], representing the $\mathit{\alpha }$-tree as a BPT by imposing the order on its edges, and using the Max-tree algorithm as the canvas for the Tree of Shapes construction algorithm [39]. Relations between some other hierarchies have also been explicated in [40], but relations between all the hierarchies examined herein are still unknown and merit further examination.

8. Conclusions

We present a unifying survey of hierarchies of partitions and partial partitions used in practice, and propose a framework which decouples the structural shape information from the scale information added by indexing. The paper further offers a unifying view on the taxonomy and indexing of image hierarchies through component trees, which correspond naturally to the way the hierarchies are constructed and explicitly define the parent–child (inclusion) information between the regions. We propose two distinct categories of image hierarchies: inclusion and partitioning trees, and identify the restrictions to the general component tree formalization implied by the categories. The formalization through component trees can also be applied to potential new hybrid classes of trees by imposing different restrictions to the general definition. An established framework for indexing the hierarchies by dendrograms was described, which allows enriching the structural representation of the hierarchy with scale information. We propose a mapping from indexed inclusion trees to indexed partitioning trees, allowing us to recover the original inclusion tree.

We have presented the details, characteristics and construction algorithms of a large number of component trees, offering an application-independent way of comparing them, which makes it the most comprehensive such study thus far. The current growing interest in processing techniques interacting with image regions or superpixels rather than individual image elements and requiring a representation extending through multiple scales, often modeled as a hierarchy, make such a paper of particular interest.