Containment Control-Guided Boundary Information for Semantic Segmentation

Abstract

Real-time semantic segmentation is a challenging task in computer vision, especially in complex scenes. In this study, a novel three-branch semantic segmentation model is designed, aiming to effectively use boundary information to improve the accuracy of semantic segmentation. The proposed model introduces the concept of containment control in a pioneering way, which treats image interior elements as well as image boundary elements as followers and leaders in containment control, respectively. Based on this, we utilize two learnable feature fusion matrices in the high-level semantic information stage of the model to quantify the fusion process of internal and boundary features. Further, we design a dedicated loss function to update the parameters of the feature fusion matrices based on the criterion of containment control, which enables fine-grained communication between target features. In addition, our model incorporates a Feature Enhancement Unit (FEU) to tackle the challenge of maximizing the utility of multi-scale features essential for semantic segmentation tasks through the meticulous reconstruction of these features. The proposed model proves effective on the publicly available Cityscapes and CamVid datasets, achieving a trade-off between effectiveness and speed.

1. Introduction

Semantic segmentation is pivotal in computer vision, facilitating comprehensive scene analysis. Among the inherent challenges faced in semantic segmentation, the precise delineation of boundaries between different categories stands out. These boundaries often imply transition regions between objects or elements with different semantics. Accurately capturing the spatial relationships and transitions between different categories enables a more nuanced and context-aware segmentation, which in turn contributes to improved scene understanding. Thus, in the field of semantic segmentation, the quality of models is intrinsically linked to their ability to recognize complex boundaries [1,2,3,4]. A model’s proficiency in accurately delineating boundaries directly affects its ability to interpret complex scenes.

Traditional methods may have difficulty in capturing subtle transitions between categories, leading to less accurate segmentation [5,6]. Effective utilization of these transition cues can significantly improve segmentation accuracy, thereby improving downstream applications such as scene understanding. Therefore, our work aims to address the limitations of traditional semantic segmentation methods by introducing an innovative approach that not only effectively utilizes boundary information but also establishes structured constraints between boundaries and internal features.

Containment control is a prevalent issue in the study of multi-agent systems (MASs), aiming to drive a subset of agents, often referred to as followers, to ultimately reside within a convex hull defined by certain specific agents, commonly known as leaders [7,8]. The development and implementation of containment control strategies integrate control theory and graph theory to address issues such as information diversion, cooperation, and self-organization in MASs [9,10,11]. However, the application of containment control should not be limited to the field of control. By transferring the relationship between leaders and followers to semantic segmentation, specifically where leaders represent boundary features and followers represent internal representations, the precise modeling of the interrelationship between boundary features and internal representations can be achieved, guiding more accurate semantic segmentation in images.

Our work not only conceptually incorporates containment control but also applies the classical theory of containment control to practically determine the spatial relationship between boundary feature information and internal feature information. This approach helps to quantify the dynamic exchange between boundary features and internal features, which leads to a nuanced control of the model utilizing boundary feature information. This pioneering integration of containment control principles into semantic segmentation models is expected to become a new paradigm in the field of visual scene understanding. Meanwhile, our network achieves high accuracy and real-time requirements as shown in Figure 1.

In conclusion, our work brings forth significant contributions, including the following:

• Boundary Feature Integration: We recognize the importance of boundary feature information in semantic segmentation tasks and dig deep into the way it is used. The dynamic fusion process of boundary features and internal representations is quantified by building a features fusion matrix based on the Laplacian matrix, which effectively improves the model’s ability to capture and utilize boundary information.
• Containment Control-Guided Segmentation: We utilize the containment control criterion to construct a dedicated loss function for updating the features fusion matrix, which achieves finer control over the information fusion between boundary features and internal representations. Through the gradient descent algorithm, on the one hand, the internal representations and the external features are made to present a contained spatial relationship, and on the other hand, a suitable features fusion matrix can be established to guide the feature fusion.
• Multi-scale Feature Enhancement: To improve the quality of features, we have designed and introduced the Feature Enhancement Unit. This innovative module is specifically crafted to tackle the challenge of maximizing the utility of multi-scale features essential for semantic segmentation tasks through the meticulous reconstruction of these features.
• Performance Evaluation: Conduct extensive experiments and performance evaluations to validate the effectiveness of our proposed approach. Compare the results with traditional segmentation methods to showcase the superiority of our model in handling complex scene structures.

2. Related Works

2.1. Semantic Segmentation with Multi-Scale Features

Since the inception of research in this field, numerous studies have concentrated on the influence of multi-scale feature information in semantic segmentation tasks. Notable examples include architectures such as PSPNet [2], the DeepLab series [4,12,13], and ICNet [3]. PP-LiteSeg [14] is a classical real-time semantic segmentation model that optimizes the traditional encoder–decoder structure. It pays much attention to the processing of multi-scale features and further proposes the Unified Attention Fusion Module and Simple Pyramid Pooling Module, both related to multi-scale features. Addressing the critical need for a balance between precision and speed, DDRNets [15] incorporate the innovative Deep Aggregation Pyramid Pooling Module (DAPPM). This module strategically enhances the effective receptive fields and fuses multi-scale context from low-resolution feature maps, contributing to the method’s exceptional performance. EMSNet [16] is a novel convolutional network for semantic segmentation that integrates an Enhanced Regional Module and a Multi-scale Convolution Module. This approach improves segmentation performance by effectively leveraging multi-scale context and pixel-to-pixel similarity in the channel direction, achieving notable results on benchmark datasets. Yan et al. [17] propose a novel multi-scale learner, Varying Window Attention, to address issues of scale inadequacy and field inactivation in semantic segmentation. Similarly, our study emphasizes the fine-grained treatment of multi-scale features. We build upon the advancements made in backbone fusion modules and introduce a novel Feature Enhancement Unit (FEU) aimed at enhancing the quality of multi-scale features. This module contributes to an improved and nuanced handling of information across different scales.

2.2. Semantic Segmentation of Geometric Relations

Several network architectures propose leveraging boundary information, achieving high accuracy while also considering inference speed. These can be summarized as follows. The authors of BNFNet [1] propose a model that integrates a fully convolutional network with boundary cues, aiming to enhance the object localization capability by leveraging boundary information. Graph-Segmenter [18] is a transformer-based semantic segmentation approach that enhances relation modeling between image regions and pixels using a graph transformer and boundary-aware attention module. AMKBANet [19] proposes an attention-based multi-kernelized and boundary-aware network that performs well in semantic segmentation tasks. CBLNet [20] provides a plug-and-play conditional boundary loss approach, which effectively improves the performance of boundary segmentation. BANet [21] adopts an encoder–decoder architecture and additionally incorporates boundary information to accurately delineate the shape and boundary of objects. AGLNet [22] introduces a global attention upsampling module to capture object shapes and boundaries. The authors of EdgeNet [23] propose a category-aware edge loss module that improves inference accuracy without compromising model inference speed. PIDNet [24] innovatively incorporates PID control methods into semantic segmentation research. The authors map proportional, integral, and derivative controllers to detail feature branches, context information branches, and boundary attention branches, achieving exceptional inference accuracy while maintaining high inference speed.

3. Method

3.1. How Containment Control Guides Semantic Segmentation

In this subsection, we will specifically introduce containment control and apply its ideas to semantic segmentation. Specifically, we consider image boundary features and internal features as “leaders” and “followers” in containment control, respectively; see Figure 2. And we use the containment control criterion to construct a unique loss function: boundary-internal containment loss function (BIC-loss). BIC-loss is used to quantify the connection between the boundary information and the internal information of the image, and to guide the updating of the weights of the features fusion matrix, which will be introduced below. Now, let us get started with the containment control.

A multi-agent system (MAS) normally consists of several leaders and followers. Containment control is a prevalent issue in the multi-agent system study for designing controllers for the followers to ultimately reside within the convex hull formed by the leaders as shown in Figure 3. The topology of interactions among agents in a MAS can be represented using a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ with a set of N nodes $\mathcal{V}=\{v_1,v_2,\dots ,v_N\}$ and a set of edges $\mathcal{E}\subseteq \mathcal{V}\times \mathcal{V}$. $v_i,\forall i\in \{1,2,\dots ,N\},$ represents the agents in the MAS, while the edges represent the communication strength between the agents. The Laplacian matrix is defined as follows:

$$\begin{array}{c}\hfill \mathcal{L}=\left\{\begin{array}{cc}-a_{ij}\hfill & i\ne j,\hfill \\ {\sum }_{j=1}^N{a}_{ij}\hfill & i=j,\hfill \end{array}\right.\end{array}$$

(1)

where $\mathcal{L}\in {\mathbb{R}}^{N\times N}$ of $\mathcal{G}$. $a_{ij}$ signifies the weights of the edges, indicating that nodes j and i are adjacent if node i can directly receive information from node j for $(v_j,v_i)\in \mathcal{E}$. In this case, $a_{ij}>0$; otherwise, $a_{ij}=0$ means that there is no direct connection between i and j, i.e., $(v_j,v_i)\notin \mathcal{E}$. $a_{ii}=0$, $\forall i\in \{1,2,\dots ,N\}$ implies the absence of self-loops. The Laplacian matrix $\mathcal{L}$ reflects the internal communication situation within the system, which inspires us to utilize it as a bridge for the fine fusion between image internal features and boundary features in semantic segmentation.

In the containment control of the MAS containing N nodes, let $\mathcal{G}$ be a topology of directed communication with $M(M<N)$ followers and $(N-M)$ leaders. Note that the leaders have no neighbors. Thus, the Laplacian matrix $\mathcal{L}$ associated with $\mathcal{G}$ can be partitioned as

$$\begin{array}{c}\hfill \mathcal{L}=\left[\begin{array}{cc}{\mathcal{L}}_1& {\mathcal{L}}_2\\ {\mathbf{0}}_{(N-M)\times M}& {\mathbf{0}}_{(N-M)\times (N-M)}\end{array}\right],\end{array}$$

(2)

in which ${\mathcal{L}}_1\in {\mathbb{R}}^{M\times M}$ and ${\mathcal{L}}_2\in {\mathbb{R}}^{M\times (N-M)}$. ${\mathcal{L}}_1$ embodies the interaction among followers, while ${\mathcal{L}}_2$ embodies the interaction between followers and leaders. It is evident in [25] that every element of $-{\mathcal{L}}_1^{-1}{\mathcal{L}}_2$ possesses a non-negative value, while each row of $-{\mathcal{L}}_1^{-1}{\mathcal{L}}_2$ exhibits a sum equal to one. This is the key to the containment control criterion, and it is with the help of the containment control criterion that we construct the link between the image boundary information and the internal information.

Now, we will briefly introduce the containment control model to elicit the containment control criterion. Consider the following global linear MAS:

$$\begin{array}{ccc}\hfill & \hfill \hfill & \hfill \dot{x_f}\left(t\right)=(I_M\otimes A)x_f\left(t\right)+(I_M\otimes B)u\left(t\right)\\ \hfill & \hfill \hfill & \hfill \dot{x_l}\left(t\right)=(I_{N-M}\otimes A)x_l\left(t\right)\end{array}$$

(3)

in which $x_f\left(t\right)\in {\mathbb{R}}^{nM}$ is the global state of followers, $x_l\left(t\right)\in {\mathbb{R}}^{n(N-M)}$ is the global state of leaders, and $u\left(t\right)\in {\mathbb{R}}^M$ is the global control input. ⊗ denotes the Kronnecker product, and $I_M$ denotes a M-dimensional identity matrix. Based on the conclusion provided in the preceding paragraph,

$$\begin{array}{c}\hfill x_f+({\mathcal{L}}_1^{-1}{\mathcal{L}}_2\otimes I_n)x_l\to 0\end{array}$$

(4)

means all followers in the MAS will ultimately reside within the convex hull formed by the leaders, thereby indicating the successful resolution of the containment control. Equation (4) is just like the containment control criterion.

In the realm of semantic segmentation, conventional feature fusion methods often grapple with fully exploiting boundary information, motivating us to introduce the groundbreaking concept of containment control to intricately model the interplay between boundary and internal information. Our novel methodology strategically incorporates containment control as a guiding principle, seamlessly integrating it into the training process of the segmentation model. We establish a robust mapping between containment control, as encapsulated by Equation (4), and the intricate feature landscape within the model. Within the context of our bespoke semantic segmentation task, a dedicated network branch adeptly extracts boundary information, aligning with $x_f$ in Equation (4). Concurrently, other branches of the network extract specific segmentation results, corresponding to $x_l$ in Equation (4). Drawing inspiration from neural architecture search principles, we introduce a set of learnable feature fusion matrices $\tilde{\mathcal{L}}$ (i.e., ${\mathcal{L}}_1^{-1}{\mathcal{L}}_2)$ to orchestrate seamless communication between internal and external feature information within the model. Utilizing the features fusion matrix-derived specific weights, each channel of the internal feature map will undergo a sophisticated weighting process involving all external feature channels. Based on the above approach, we introduce a relevant loss function. Through linear neural mapping, different features extracted by our network are collapsed into one dimension. Then, the internal information ($\mathcal{I}$) and boundary information ($\mathcal{B}$) are stacked as column vectors. Using the containment control criterion (4) and the features fusion matrix $\tilde{\mathcal{L}}$, the loss function is formulated as follows:

$$\begin{array}{c}\hfill Los{s}_{cc}={\displaystyle \frac{1}{C}}\sum _{i=1}^C{(\mathcal{I}+\tilde{\mathcal{L}}\mathcal{B})}_i,\end{array}$$

(5)

in which C denotes the number of feature channels, and ${(\mathcal{I}+\tilde{\mathcal{L}}\mathcal{B})}_i$ denotes the i-th element of the vector $(\mathcal{I}+\tilde{\mathcal{L}}\mathcal{B})$. Here, we employ the term “internal information” as a comprehensive representation of detailed features and contextual features. Subsequently, we will provide a detailed exposition on this matter. The loss calculation based on the condensed information reduces the computational complexity while preserving feature information.

When we consider image boundary features and internal features as “leaders” and “followers” in containment control, it should be noted that $L_1$ in Equation (2) reflects the connection between the internal information of the image, while $L_2$ reflects the connection between the image boundary information and the internal information. Since the internal information of the image has already been interpreted by the detail features and the contextual feature branches, the spatial constraints between the internal information of the image are not taken into account when parsing the relationship between the image boundary and the internal information by using the containment control criterion. Thus, $L_1$, which reflects the connection between the internal information of the image, is set as a unit matrix to avoid the model interpreting the internal features of the image again and to improve the efficiency of the model inference. In addition, since BIC-loss is calculated using one-dimensional information after linear neural mapping, $I_n$ in Equation (4) is actually equal to one.

We visualize the construction process of the loss function in Figure 3. Stage 1 represents feature compression. Stage 2 is the linear neural mapping used to extract more representational features. Stage 3 is the computation process for $Los{s}_{cc}$. Stage 4 indicates the updating basis of the features fusion matrix $\tilde{\mathcal{L}}$.

3.2. CcNet

In this work, we propose the Containment Control Network (CcNet) to improve the accuracy of complex semantic segmentation tasks by seamlessly integrating the containment control into the model as shown in Figure 4. CcNet adopts PIDNet as the backbone and is composed of three branches: the detailed branch (D), the contextual branch (C), and the boundary branch (B). The design of CcNet places particular emphasis on the relationships between the boundary features and internal features, leveraging containment control mechanisms. This strategic approach facilitates the improved integration of boundary information from the B branch into the internal features of the D and C branches.

To achieve a nuanced fusion of internal and boundary feature information, we introduce a Feature-Weighted Fusion (FWF) mechanism. This mechanism ensures that each channel of the internal feature incorporates information from the boundary feature, represented as the weighted sum of all channels of the boundary feature. The features fusion matrix $\tilde{\mathcal{L}}$ is specifically designed to quantify the compression process applied to the boundary feature, tailoring it to provide customized boundary features for the internal features. ${\tilde{\mathcal{L}}}_1$ plays a crucial role in guiding the fusion between the D branch and the B branch, facilitating the amalgamation of detailed feature information and boundary feature information. Similarly, ${\tilde{\mathcal{L}}}_2$ is instrumental in guiding the fusion of the C branch and the D branch, promoting the integration of contextual feature information and boundary feature information. Both feature fusion matrices have corresponding $Los{s}_{cc}$ terms dedicated to their updates. These loss terms help ensure the effective utilization of boundary feature information by regulating the fusion process in their respective branches.

In the FWF mechanism, we first weight the boundary feature channels, i.e.,

$$\begin{array}{c}\hfill {\mathcal{B}}_i\left(j\right)=\tilde{\mathcal{L}}(i,j)\times \mathcal{B}\left(j\right),\end{array}$$

(6)

in which $\mathcal{B}\left(j\right)$ denotes the $j^{th}$ channel feature map of the boundary feature channel, and $\tilde{\mathcal{L}}(i,j)$ denotes the value of the ${(i,j)}^{th}$ element of the feature fusion matrix. Then, the weighted boundary feature channel is compressed and downsized, and summed with the internal feature channel to obtain the output, i.e.,

$$\begin{array}{c}\hfill out\left(i\right)=\sum _{j=1}^C{\mathcal{B}}_i\left(j\right)+\mathcal{I}\left(i\right),\end{array}$$

(7)

in which C denotes the number of feature channels. The sum over channels ${\sum }_{j=1}^C{\mathcal{B}}_i\left(j\right)$ is akin to a 1 × 1 convolution, which is essentially a weighted sum across the channels. Therefore, the FWF mechanism, with its complexity of $C\times H\times W\times C$, is a very efficient operation, especially given the substantial functionality it provides in processing feature maps. This efficiency makes it well suited for scenarios where both computational cost and performance are critical.

3.3. Feature Enhancement Unit

To enhance the quality of multi-scale features in semantic segmentation models, we design a Feature Enhancement Unit (FEU) as shown in Figure 5. This module draws inspiration from classical feature enhancement models [26,27] and adopts a “split-restructure” paradigm.

In the splitting phase, each channel of the input feature map is associated with a set of learnable parameters $\alpha$ in the Group Normalization (GN) layer. On the one hand, SoftMax operations on $\alpha$ produce the parameter $\beta$, from which two different masks $M_1$ and $M_2$ are derived according to a predetermined threshold. These masks help to split the spatial locations of representation information and redundant feature information. On the other hand, the feature maps that have gone through the GN layer are multiplied with the parameter $\beta$ and activated by a Sigmoid function to produce new weights for reweighting. Subsequently, the new weighting parameters together with the two masks will generate two separated feature maps, i.e., crucial representation information and redundant feature information. In the merge phase, the feature information considered important by the model is mined more deeply so that more computation is used on the important features. To achieve this, we employ group-wise and point-wise processing for crucial representations, while adopting only point-wise processing for redundant features. Next, a dedicated compression module is designed to compress each channel within the two feature information streams. This module excels at optimizing the spatial structure of features, taking into account the spatial relationships between adjacent channels when generating attention weights. These weights undergo competition at corresponding locations before being transmitted back to their respective feature information streams for weighted processing. The final step in this process involves directly merging the two feature information streams, preventing the loss of important information.

4. Experiment

4.1. Dataset

Cityscapes

Cityscapes [28] comprises 5000 meticulously labeled images captured from 50 different cityscapes. The images are thoughtfully partitioned into sets of 2975, 500, and 1525 for training, validation, and testing, respectively. Since the test set of Cityscapes is not publicly available, we used the validation set for evaluation. With an image resolution of 2048 × 1024, the dataset poses a considerable challenge to the model.

CamVid

Cambridge-driving Labeled Video Database (CamVid) [29] comprises 701 driving scenes, meticulously divided into sets of 367, 101, and 233 images for training, validation, and testing, respectively. The images have a resolution of 960 × 720, and annotations span across 32 categories. Notably, 11 of these categories have been deliberately chosen to ensure a fair and meaningful comparison with prior studies.

4.2. Quantitative Analysis

Cityscapes

Our quantitative analysis focuses on evaluating the performance of CcNet against state-of-the-art semantic segmentation models using the Cityscape dataset.

Table 1. Comparison of accuracy and speed on Cityscapes.

Model	GPU	Resolution	mIOU (%)	#FPS	GFLOPs	Params (M)
CABiNet [30]	GTX 2080Ti	2048 × 1024	76.6	76.5	12.0	2.64
AMKBANet-T [19]	A100	2048 × 1024	81.1	-	884	64
BiSeNetV2 [5]	GTX 1080Ti	1536 × 768	74.8	65.5	55.3	49
BiSeNetV2-L [5]	GTX 1080Ti	1024 × 512	75.8	47.3	118.5	-
STDC1-Seg75 [6]	RTX 3090	1536 × 768	74.5	74.8	-	-
STDC2-Seg75 [6]	RTX 3090	1536 × 768	77.0	58.2	-	-
PP-LiteSeg-T2 [14]	RTX 3090	1536 × 768	76.0	96.0	-	-
PP-LiteSeg-B2 [14]	RTX 3090	1536 × 768	78.2	68.2	-	-
HyperSeg-M [31]	RTX 3090	1024 × 512	76.2	59.1	7.5	10.1
HyperSeg-S [31]	RTX 3090	1536 × 768	78.2	45.7	17.0	10.2
DDRNet-23-S [15]	RTX 3090	2048 × 1024	77.8	108.1	36.3	5.7
DDRNet-23 [15]	RTX 3090	2048 × 1024	79.5	51.4	143.1	20.1
MRFNet-S [32]	GTX 1080Ti	512 × 1024	79.3	144.5	-	-
MRFNet-L [32]	GTX 1080Ti	1024 × 1024	79.9	73.63	-	-
PIDNet-S-Simple [24]	RTX 3090	2048 × 1024	78.8	100.8	46.3	7.6
PIDNet-S [24]	RTX 4090	2048 × 1024	78.8	93.2	47.6	7.6
PIDNet-M [24]	RTX 3090	2048 × 1024	80.1	39.8	197.4	34.4
PIDNet-L [24]	RTX 4090	2048 × 1024	80.9	31.1	275.8	36.9
CcNet-S	RTX 4090	2048 × 1024	78.5	83.4	55.0	7.85
CcNet-L	RTX 4090	2048 × 1024	81.2	30.2	290.2	37.4

provides a comprehensive overview of the performance of various models, including CcNet, on different metrics. CcNet exhibits very competitive performance, with mIOUs of 78.5% and 81.2% for the small variant (CcNet-S) and the large variant (CcNet-L). In comparison with AMKBANet-T, CcNet-L attains superior performance while featuring a smaller parameter size. These results demonstrate the efficacy of the proposed models in achieving high segmentation accuracy. CcNet-S with 83.4 FPS and CcNet-L with 30.2 FPS belong to the real-time semantic segmentation models. CcNet shows good results in different sizes of models. CcNet performs well in models of different sizes and always maintains a very competitive performance. In conclusion, our quantitative analysis emphasizes the strong performance of CcNet in semantic segmentation tasks. This is strong evidence that containment control can have a positive impact on the results of semantic segmentation tasks.

CamVid

Our quantitative evaluation is centered on assessing the efficacy of CcNet in comparison to state-of-the-art semantic segmentation models, employing the CamVid dataset as depicted in

Table 2. Comparison of accuracy and speed on CamVid.

Model	mIOU	#FPS	GPU
MSFNet [33]	75.4	91.0	GTX 2080Ti
PP-LiteSeg-T [14]	75.0	154.8	GTX 1080Ti
TD2-PSP50 [34]	76.0	11.0	TITAN X
BiSeNetV2 [5]	76.7	124.0	GTX 1080Ti
BiSeNetV2-L [5]	78.5	33.0	GTX 1080Ti
HyperSeg-S [31]	78.4	38.0	GTX 1080Ti
HyperSeg-L [31]	79.1	16.6	GTX 1080Ti
TSA-LNet [35]	79.7	143.0	GTX 2080Ti
DDRNet-23-S [15]	78.6	182.4	RTX 3090
DDRNet-23 [15]	80.6	116.8	RTX 3090
PIDNet-S [24]	80.1	153.7	RTX 3090
PIDNet-S-Wider [24]	82.0	85.6	RTX 3090
CcNet-s	80.6	182.0	RTX 4090

. In terms of mIOU, CcNet emerges as the superior performer, achieving an impressive score of 80.6%. While CcNet shares a similar scale with PIDNet-S, its performance does not surpass that of PIDNet-S-Wider; however, CcNet demonstrates faster inference. Additionally, CcNet-s attains an impressive rate of 182 frames per second on a RTX 4090. This underscores the suitability of the overall CcNet model for GPU computations, with the inference speed exhibiting a notable dependence on the scale of the input image.

4.3. Qualitative Analysis

Our network places special emphasis on the constraint of boundary information on internal information, resulting in more detailed delineation for classification. The qualitative analysis in Figure 6 highlights the superiority of our approach. In the first two images, our proposed CcNet demonstrates its superior performance by accurately identifying and segmenting small distant objects based on boundary features. For example, in the second image, traffic signs are located to the left of both the car and the second tree. The second traffic signpost on the left blends into the background due to its similar color, resulting in its omission in the PIDNet segmentation output. Although these signs are not annotated in the ground truth, our model effectively detects them, highlighting its robustness in segmenting subtle features. In the third image, multiple overlapping bicycles are closely positioned behind a truck. In this scenario, PIDNet generates a segmentation result that abruptly segments the bicycles, failing to capture their continuity and spatial relationships. This limitation indicates the difficulty of PIDNet in maintaining consistency in scenarios involving overlapping objects. In contrast, CcNet produces higher-quality segmentation results by effectively preserving object continuity and spatial context. The fourth image illustrates the limitations of PIDNet in correctly interpreting the seat portions of bicycles, which are misclassified as pedestrians. This misclassification underscores the challenges of PIDNet in distinguishing between similar objects and fine-grained details within complex scenes, while CcNet demonstrates improved capability in leveraging boundary features for accurate classification. In the fifth image, PIDNet overlooks certain well-defined classes, exposing its shortcomings in boundary delineation and class representation. In the sixth image, CcNet significantly enhances semantic information retrieval, particularly for distant fences; its ability to provide detailed semantic information for objects at varying distances reflects its advanced boundary recognition capabilities and overall robustness in semantic segmentation tasks. Overall, the results indicate that CcNet more effectively utilizes boundary information, providing more accurate and detailed semantic segmentation in complex traffic scenarios compared to PIDNet.

Figure 7 demonstrate that after weighting through the FWF module, the boundary information achieves desirable constraint effects on both the detailed branch and the contextual branch. The boundaries between different categories are adequately delineated, further confirming the effectiveness of our approach.

4.4. Ablation Study

We perform ablation experiments on two variants of CcNet, named CcNet-S and CcNet-L.

Features fusion matrix $\tilde{\mathcal{L}}$

The features fusion matrix $\tilde{\mathcal{L}}$ serves as a quantifying tool for the fusion process between boundary and internal features. In this context, we focus on a comprehensive analysis of the impact of different configurations on the features fusion matrix within the CcNet architecture. To establish targeted communication channels between internal and boundary information, we introduce two sets of matrices. Two distinct losses guide the fusion of the D branch, C branch, and B branch, necessitating mappings from two sets of internal semantic information. Thus, for each variant of CcNet, we investigate the influence of three different combinations of internal feature information on the Laplacian matrix: D branch, C branch, and Head.

Experimental results are shown in

Table 3. Ablation study of FWF.

Model	FWF			mIOU
	D Branch	C Branch	Head
CcNet-S		✔	✔	78.12
	✔		✔	78.16
	✔	✔		78.48
CcNet-L		✔	✔	79.85
	✔		✔	80.55
	✔	✔		81.15

, indicating consistent effects across both variants of CcNet for different combinations of feature information. The most effective approach involves mapping features from the D branch and C branch, utilizing the generated loss to guide the fusion of features between the self-branch and the B branch. This targeted configuration demonstrates optimal efficiency in leveraging the Laplacian matrix for enhanced fusion of internal and boundary features. These findings provide valuable insights into the role of the features fusion matrix in guiding feature fusion processes within CcNet.

FEU

This study aims to assess the performance of the FEU module concerning its placement and to analyze its influence on the two variants of CcNet. The results are shown in

Table 4. Ablation study of FEU for CcNet.

Model	FEU		mIOU
	C Branch	Head
CcNet-S			77.78
	✔		78.04
		✔	78.48
CcNet-L			80.73
	✔		81.15
		✔	80.00

. There are two key variations to the location of the FEU module: the C branch and the Head. The C branch processes contextual information, while the Head processes the final feature information fed to the Head of the network. In the context of different variants, CcNet-S and CcNet-L, the introduction of the C branch into the FEU module significantly improves mIOU. This enhancement is attributed to the inherent rich contextual information of the C branch, which undergoes further optimization through the DAPPM module. Hence, the demand for optimizing multi-scale features at this juncture is particularly pronounced.

The structure of the FEU is quite complex. Therefore, to more intuitively demonstrate the complexity of the FEU, we conduct comparative experiments based on CcNet-s. Multiple experiments show that CcNet-s can achieve a 5 FPS improvement after removing FEU. This complexity may be due to the additional operations required during the “split-restructure” process, which improves feature quality but increases the computational load. However, the FEU can be viewed as a mechanism to balance between higher accuracy (through better feature enhancement) and faster inference speed. By including or excluding the FEU, one can adjust the performance of CcNet depending on the specific application requirements.

Additionally, relocating the FEU module from the C branch to the head in the CcNet-S variant raises the mIOU from 78.04 to 78.48. Conversely, for the CcNet-L variant, relocating the FEU module from the C branch to the head results in a sharp mIOU decline to 80.0. This indicates that, as the model size increases, the FEU module processing the results from the FWF module may have a detrimental effect. This phenomenon could be attributed to the high-quality feature maps generated by the FWF module when the model size increases, causing the FEU module to destroy the spatial relationships in the original feature maps. In summary, the FEU module enhances the quality of multi-scale features but exhibits sensitivity to its position within the model.

The loss function $Los{s}_{cc}$

We validate the effectiveness of the proposed $Los{s}_{cc}$ by performing comparative analyses with the use of random search (rs) methods. Based on the optimal model weights obtained from training, we perform three rounds of random search on $\tilde{\mathcal{L}}$ and evaluate it on the validation dataset as shown in

Table 5. Ablation study on the training methodology of $\tilde{\mathcal{L}}$.

Weights of $\tilde{\mathcal{L}}$		mIOU
Random Search	D + C
✔		65.62
✔		70.52
✔		69.17
	✔	81.15

. It can be seen that the updated model based on $Los{s}_{cc}$ achieves a mIOU value of 81.2 on the validation set, which enables the model to find a suitable features fusion matrix. This result suggests that $Los{s}_{cc}$ can effectively guide the updating of the Laplace matrix to achieve a fine fusion of boundary features and interior features in the model.

5. Conclusions

In this paper, we proposed a novel real-time three-branch semantic segmentation paradigm that leverages boundary information to enhance segmentation accuracy in real-time complex scenes. The new paradigm introduces containment control, which regards the inner and boundary elements of images as followers and leaders, respectively, so as to skillfully integrate details information, context information, and boundary information. Additionally, a specialized loss function $Los{s}_{cc}$ and the FEU for multi-scale feature reconstruction further optimize the model’s performance. Our experimental results on the Cityscapes and CamVid datasets demonstrate the model’s effectiveness, achieving a balance between accuracy and speed. These results prove the potential of using boundary information effectively in complex scenes and provide new ideas for the further development of real-time semantic segmentation technology in the future.

Notably, the features from the boundary branch need to be propagated to the other branches, highlighting the model’s heavy reliance on the quality of edge information. When the quality of the edge information is suboptimal, the Laplacian matrix $\mathcal{L}$, which is influenced by the $Los{s}_{cc}$, can cause instability in the model’s performance. Furthermore, the necessity of a dedicated branch for edge information extraction results in a relatively large model structure. This added complexity presents challenges for enhancing inference speed.