A Cross-Modal Feature Fusion Model Based on ConvNeXt for RGB-D Semantic Segmentation

Abstract

Semantic segmentation, as the pixel level classification with dividing an image into multiple blocks based on the similarities and differences of categories (i.e., assigning each pixel in the image to a class label), is an important task in computer vision. Combining RGB and Depth information can improve the performance of semantic segmentation. However, there is still a problem of the way to deeply integrate RGB and Depth. In this paper, we propose a cross-modal feature fusion RGB-D semantic segmentation model based on ConvNeXt, which uses ConvNeXt as the skeleton network and embeds a cross-modal feature fusion module (CMFFM). The CMFFM designs feature channel-wise and spectral-wise fusion, which can realize the deeply feature fusion of RGB and Depth. The in-depth multi-modal feature fusion in multiple stages improves the performance of the model. Experiments are performed on the public dataset of SUN-RGBD, showing the best segmentation by our proposed model ConvNeXt-CMFFM with the highest mIoU score of 53.5% among the nine comparative models. The outstanding performance of ConvNeXt-CMFFM is also achieved on our self-built dataset of RICE-RGBD with the highest mIoU score and pixel accuracy among the three comparative datasets. The ablation experiment on our rice dataset shows that compared with ConvNeXt (without CMFFM), the mIoU score of ConvNext-CMFFM is increased from 71.5% to 74.8% and its pixel accuracy is increased from 86.2% to 88.3%, indicating the effectiveness of the added feature fusion module in improving segmentation performance. This study shows the feasibility of the practical application of the proposed model in agriculture.

1. Introduction

Semantic segmentation is an important task in computer vision and its purpose is to divide the input image into multiple regions with coherent semantic meaning to complete pixel-dense scene understanding for many real-world applications, such as autonomous driving [1], robot navigation [2] and so on. In recent years, with the rapid development of deep learning [3,4,5,6,7], pixel-based semantic segmentation of RGB images has received more and more attention and has achieved remarkable progress in segmentation accuracy [6,7]. However, due to the characteristics of RGB images, current deep semantic segmentation models cannot always extract correct features in some specific cases. For example, when two objects have similar colors or textures, it is hard to differentiate between them through pure RGB image. In order to solve these problems, some researchers use additional information to assist in semantic segmentation.

In recent years, with the rapid development of RGB-D sensors, in addition to RGB information, Depth information can also be acquired. Depth data can show the structure and geometric information of objects in the scene and can be used as supplementary data for the simultaneous RGB data so as to obtain richer features such as color, texture and shape, and improve the accuracy of semantic segmentation. Many works have proved that spatial information is very helpful for improving the accuracy of semantic segmentation, and affirmed the effectiveness of learning from complementary patterns. With the rapid development of convolutional neural networks (CNN or ConvNet), researchers have proposed various CNN-based methods to use Depth information for RGB-D segmentation. In segmentation tasks, two mainstream designs have been widely used, namely single-stream design and two-stream design. In a single-stream design, Depth information is simply spliced directly with RGB at the input end to form a 4-channel (RGB-D) input or a 6-channel (RGB-HHA where HHA is encoded from Depth referring to dispersion, height above ground and normal angle) input, and then a single CNN module is used for further processing. However, RGB and Depth are fundamentally different. RGB values capture photometric appearance attributes in projected image space, while Depth represents geometric attributes. Although adjacent pixels are close to each other on the image plane, they are not necessarily geometrically coherent in a 3D space. Therefore, simply stitching RGB and Depth cannot fully explore the correlation between RGB and Depth images. In a dual stream design, the architecture uses parallel encoders, and RGB and Depth are processed using separate streams. However, most studies have focused on designing frameworks for processing RGB and Depth images, ignoring the complementarity of features between RGB and Depth, resulting in excessive reliance on individual learning streams, resulting in the increase in computational costs. Of course, this also leads to insufficient feature fusion between RGB and Depth images, resulting in low segmentation accuracy. In this article, considering early feature fusion and supplementation of RGB and Depth images, we propose a cross-modal feature fusion RGB-D semantic segmentation model based on ConvNeXt [8]. By adding cross-modal feature fusion modules after different levels, more sufficient complementarity and fusion of RGB and Depth features were achieved and the accuracy of RGB-D semantic segmentation was improved. The main contributions of this article are as follows.

(1) A cross-modal RGB feature and deep feature fusion module is proposed. Through cross-modal information interaction, the generalization ability of the model is improved, and the inference ability of the model is also improved through the cross-attention mechanism.

(2) An RGB-D semantic segmentation model based on ConvNext’s parallel dual two-branch structure is constructed, which can maintain the strong feature extraction capabilities of the RGB and Depth branches by cross-modal feature fusion and effectively integrate and fuse RGB and Depth features. This model exhibits better segmentation performance for large datasets.

2. Related Work

In earlier studies, researchers manually customized the fusion features of RGB-D. In recent years, RGB-D semantic segmentation methods based on deep learning have dominated the mainstream due to the great advantage of deep learning in the ability to extract features [9,10,11,12,13,14,15,16,17,18,19,20,21,22]. ACNet [9] proposed a channel attention module to fuse RGB features and Depth features. The authors of references [10,11] used three channels of horizontal parallax, ground height, angle and gravity to HHA encode Depth images, and this method of processing Depth information has been widely used in later algorithms. FuseNet [12] introduced a fuse layer to embed Depth features into RGB features. The authors of references [13,14] proposed an efficient feature fusion module for objects containing different levels of information by adopting multimodal feature fusion and multi-level feature refinement to capture RGB-D features. LSD-GF [15] introduced a gated fusion layer to adjust the RGB and Depth contributions on each pixel. Depth-aware convolution and pooling were achieved by integrating geometric information into RGB features [16]. CFN [17] utilized Depth information to segment an image into layers representing similar visual features. SCN [18] utilized Depth data to flexibly select useful contextual information for image regions where different objects coexist. J. McCormac et al. [19] superimposed RGB and Depth features into four channels to improve semantic segmentation.

The two-stream structures became the mainstream framework for future RGB-D semantic segmentation due to the high efficiency and variability. Qi et al. [20] introduced a 3D graph neural network to model accurate context through geometric cues provided by the Depth data. Zhang et al. [21] proposed a novel Task-Recursive Learning (TRL) framework to jointly and recurrently conduct three representative tasks containing depth estimation, surface normal prediction and semantic segmentation. Zhou et al. [22] proposed a pattern-structure diffusion (PSD) framework to mine and propagate task-specific and task-across pattern structures in the task-level space for joint depth estimation, segmentation and surface normal prediction. Since RGB-D contains the information of two different modes, the fusion of RGB and Depth information becomes an effective method to improve the accuracy of semantic segmentation [23,24,25,26,27,28]. Fan et al. [23] constructed an encoder network with two ConvNext-T backplates for each of RGB and Depth, and a decoder network composed of multi-scale supervision and multi-granularity segmentation branches to achieve scene segmentation at different scales. Yang et al. [24] proposed a new framework, MGCNet, that guides the fusion of patterns through differential exploration to reduce collaborative conflicts. In the decoder, a gating feature was proposed to avoid the exclusion of inter-layer information and capture contextual information adequately. Bai et al. [28] proposed a two-branch network called the differential convolution attention network (DCANet), which composed of a pixel differential convolution attention and a set differential convolution attention, and was used to fuse local and global information of two-mode data. Wu et al. [29] proposed a new framework for integrating Depth information into RGB CNN to guide feature extraction on RGB images. Some researchers focus on 3D scene completion, using deep learning methods and RGB-D data to achieve semantic segmentation and complete of three-dimensional scenes [30,31,32]. These semantic segmentation networks open up new ways to accurately extract Depth information. However, the difficult problem of ways to fully integrate RGB-D information persist. We can conclude that the key challenge of RGB-D semantic segmentation is how to identify the difference between RGB features and Depth features and integrate them effectively and use them to achieve higher segmentation accuracy.

3. Method

3.1. Framework Overview

In this paper, a cross-modal feature fusion RGB-D semantic segmentation model based on ConvNeXt is proposed. The framework of the model is shown in Figure 1. We employ two parallel RGB branches and a Depth branch to extract features from RGB and Depth images. At the same time, the cross-modal feature fusion module is used to supplement the features of RGB and Depth branch, that is, the features of different modalities are supplemented by this module, and then the supplemented features are fused to achieve cross-modal feature fusion.

The encoder is used to extract RGB and Depth features at different levels, and then the decoder is used to convert the feature maps at different levels into the final semantic map. In order to improve the special utilization of different levels, we introduce multi-level feature supplement in the decoder and use the fused features in different levels in the encoder as the supplement to improve the robustness of the model. The model encoder consists of 4 stages. Later, cross-modal feature fusion module (CMFFM), which is presented in Section 3.3, is embedded in each stage, the RGB features and Depth features are sent to the next layer through the CMFFM, and the fusion features of RGB and Depth are sent to the decoder for feature supplementation. In Figure 1, the functions of downsampling and upsampling are to resize the image size. For example, the “1/4” in encoder and decoder part means the spatial size is reduced and enlarged to 1/4 and 4 times of the original size, respectively.

3.2. ConvNeXt

Since the proposal of VIT [33], it has rapidly replaced the convolutional network as the state-of-the-art image classification model. Using Transformer as the backbone network and introducing convolutional neural networks (ConvNet) enables Transformer to be applied in a variety of visual tasks, such as object detection, semantic segmentation, etc. ConvNeXt builds a network entirely composed of standard ConvNet modules based on the design of VIT and ResNet, which is superior to Transformer in accuracy and scalability while maintaining the simplicity and efficiency of standard ConvNet. The network structure of ConvNeXt consists of four layers shown in Figure 2. Layers 1, 2, and 4 contain three basic blocks, and Layer 3 contains 27 basic blocks. Each basic block contains three convolutional layers, and the Gaussian Error Linear Unit (GELU) [34] activation function and the simpler Layer Normalization (LN) [35] are used in each basic block. RGB image and Depth image are input through their respective branches in the model (Figure 1). After passing through the 1st downsampling layer, the RGB data and Depth data have the same data shape with 192 channels through the convolution operation (in Figure 2), and then the 192-channelled data are sent to Layer 1 for processing, and after the fusion module (i.e., CMFFM) of Layer 1, the data is sent to Layer 2 (for further downsampling) and the decoder modular (for upsampling with feature supplement) at the same time, then to Layers 3 and 4. Each Layer is connected by the downsampling layer.

3.3. Cross-Modal Feature Fusion Module (CMFFM)

We propose a cross-modal feature fusion module (CMFFM) to fuse RGB and Depth features, as shown in Figure 3. In order to improve the extraction and fusion capabilities of the multimodal features of the model, CMFFM processes the input data from RGB and depth features. For the input RGB feature, $F_{RGB}\in {\mathbb{R}}^{H\times W\times C}$ and the Depth feature $F_{Depth}\in {\mathbb{R}}^{H\times W\times C}$, global max pooling and global average pooling are adopted along the channel-wise phase to retain more information, and finally four result vectors are obtained. Connecting vectors $Y_{RGB}\in {\mathbb{R}}^{2C}$ and $Y_{Depth}\in {\mathbb{R}}^{2C}$ are obtained by the RGB and the Depth feature, respectively. After MLP operations on $Y_{RGB}$ and $Y_{Depth},$ respectively, the weights $W_{RGB}^C\in {\mathbb{R}}^{2C}$ and $W_{Depth}^C\in {\mathbb{R}}^{2C}$ are obtained through the Sigmoid function to split $W_{RGB}^C$ into $W_{RGB}^{C1}\in {\mathbb{R}}^C$ and $W_{RGB}^{C2}\in {\mathbb{R}}^C$ and split $W_{Depth}^C$ into $W_{Depth}^{C1}\in {\mathbb{R}}^C$ and $W_{Depth}^{C2}\in {\mathbb{R}}^C$.

$$\genfrac{}{}{0pt}{}{W_{RGB}^{C1},W_{RGB}^{C2}={\mathrm{F}}_{split}\left(Sigmoid\left(MLP\left(Y_{RGB}\right)\right)\right)}{W_{Depth}^{C1},W_{Depth}^{C2}={\mathrm{F}}_{split}\left(Sigmoid\left(MLP\left(Y_{Depth}\right)\right)\right)},$$

(1)

where $F_{split}$ means to split a vector into two vectors. The weights $F_{RGB}^C\in {\mathbb{R}}^{H\times W\times C}$ and $F_{Depth}^C\in {\mathbb{R}}^{H\times W\times C}$ of the RGB modality and the Depth modality in the channel-wise phase are calculated by the following formulas:

$$\genfrac{}{}{0pt}{}{F_{RGB}^C={{\lambda }_{C1}W}_{RGB}^{C1}\ast F_{RGB}+{\lambda }_{C2}{W}_{RGB}^{C2}\ast F_{Depth}}{F_{Depth}^C={{\lambda }_{C1}W}_{Depth}^{C1}\ast F_{Depth}+{{\lambda }_{C2}W}_{Depth}^{C2}\ast F_{RGB}},$$

(2)

where ∗ represents multiplication, ${\lambda }_{C1}$ and ${\lambda }_{C2}$ are hyperparameters, which are both set to 0.5 in this paper. In the spatial-wise phase, $F_{RGB}$ and $F_{Depth}$ are connected, and after two convolution layers with a convolution kernel of 1 × 1 and a RELU function, the Sigmoid function is used to obtain the feature map $Z\in {\mathbb{R}}^{H\times W\times 2}$, and then split it into two weight maps $W_{RGB}^S$ and $W_{Depth}^S$. The calculation formulas for the weights $F_{RGB}^S\in {\mathbb{R}}^{H\times W\times C}$ and $F_{Depth}^S\in {\mathbb{R}}^{H\times W\times C}$ in the spatial-wise phase are calculated as follows:

$$Z={Conv}_{1\times 1}\left(RELU\left({Conv}_{1\times 1}\left(F_{RGB}\left|\right|F_{Depth}\right)\right)\right),$$

(3)

$$W_{RGB}^S,W_{Depth}^S={\mathrm{F}}_{split}\left(Sigmoid\left(Z\right)\right),$$

(4)

$$\genfrac{}{}{0pt}{}{F_{RGB}^S=W_{RGB}^S\ast F_{RGB}}{F_{Depth}^S=W_{Depth}^S\ast F_{Depth}},$$

(5)

where |.| represents the connection operation, ${Conv}_{1\times 1}$ is the convolution operation with the convolution kernel of 1 × 1. The final RGB feature output $F_{RGB}^{out}\in {\mathbb{R}}^{H\times W\times C}$, Depth feature output $F_{Depth}^{out}\in {\mathbb{R}}^{H\times W\times C}$ and fusion feature output $F_{Fusion}\in {\mathbb{R}}^{H\times W\times C}$ are calculated as follows:

$$\genfrac{}{}{0pt}{}{F_{RGB}^{out}=F_{RGB}+{\lambda }_C{F}_{RGB}^C+{\lambda }_S{F}_{RGB}^S}{F_{Depth}^{out}=F_{Depth}+{\lambda }_C{F}_{Depth}^C+{\lambda }_S{F}_{DEpth}^S},$$

(6)

$$F_{Fusion}={Conv}_{1\times 1}\left(RELU\left({Conv}_{1\times 1}\left(F_{RGB}^{out}\left|\right|F_{Depth}^{out}\right)\right)\right)+{Conv}_{1\times 1}\left(F_{RGB}^{out}\left|\right|F_{Depth}^{out}\right).$$

(7)

After CMFFM, $F_{RGB}^{out}$ and $F_{Depth}^{out}$ are used as the input of the next layer again, and $F_{Fusion}$ is sent to the decoder as a supplementary feature of the features of different levels in the decoder.

4. Results

4.1. Experimental Parameters and Evaluation Indexes

Because the deep neural network training process has many iterations and a large number of matrix operations and requires a large amount of computing resources, the high-performance graphics processing unit (GPUs) is indispensable. In this experiment, NVIDIA GeForce RTX 3090 is used with graphics memory of 24 GB. The CPU model is Intel(R) Core (TM) i9-10900K (3.70 GHz) with the memory size of 128 GB. The operating system is Ubuntu20.04, and the model was implemented using PyTorch deep learning framework, while CUDA Toolkit 11.1 and CUDNN V8.0.4 are used for computation acceleration. Anaconda3.6 and Python are used as the development environment and programming language for the model. In our model, the weights for all layers of the network are initialized to a commonly normal distribution with the mean of 0, the variance of 0.01 and the deviation of 0. The two parameters ${\lambda }_{C1}$ and ${\lambda }_{C2}$ in Equation (2) are both initialized to 0.5. The two parameters ${\lambda }_C$ and ${\lambda }_S$ of Equation (6) are both initialized to 0.5. In model training, two public datasets (NYUDv2 and SUN-RGBD) and a self-built Rice-RGB-D dataset are used. For public datasets, the size of the images, which are input to the model, is $480\times 640$, and for the rice dataset, the size is $512\times 160$. Adam is used as the optimizer with a learning rate of 2 × 10⁻⁵ and weight decay of 5 × 10⁻⁴. We adopt a poly learning rate schedule with factor ${\left(1-iter/{iter}_{max}\right)}^{0.9}$ and use cross-entropy as the loss function, which is defined as

$$Loss=-\frac{1}{n}\sum _x\left[y\mathrm{l}\mathrm{n}{y}^{\prime }+\left(1-y\right)\ln \left(1-y^{\prime }\right)\right],$$

(8)

where y and y′ denote the expected and actual output. We use the batch size of eight and epoch of 200 to train on all datasets. Experimental operation parameters are shown in

Table 1. Running parameter table.

Parameter		Values
Operating system		Ubuntu20.04
CPU		Intel(R) Core (TM) i9-10900K (3.70 GHz)
GPU		GeForce RTX 3090
Development environment		Anaconda3.6
Framework		Pytorch1.8
Input size	NYUDv2	480 × 640
	SUN-RGBD	480 × 640
	RICE-RGBD	512 × 160
Learning Rate		2 × 10−5
Batch size		8
Epoch		200

.

To evaluate the performance of the different methods, we use prevailing Pixel Accuracy (Pixel Acc.) and mean Intersection over Union (mIoU) as evaluation indicators, which are defined as [36]

$$Pixel Acc.=\sum _i\frac{n_{ii}}{s},$$

(9)

$$mIoU=\frac{1}{n_c}\sum _i\frac{n_{ii}}{{(s}_i-n_{ii}+\sum _j{n}_{ji})},$$

(10)

where $n_{ji}$ is the number of pixels with ground-truth class $j$ predicted as class $i (\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{n} j \mathrm{e}\mathrm{q}\mathrm{u}\mathrm{a}\mathrm{l}\mathrm{s} \mathrm{t}\mathrm{o} i,n_{ji}=n_{ii})$, $n_c$ is the total number of classes, $s_i$ is the number of pixels with ground truth class $i$, and $s$ is the total number of all pixels.

4.2. Public Datasets

We conduct experiments on two public benchmark datasets: NYUDv2 [37], SUN-RGBD [38]. The NYUDv2 dataset contains 1449 RGB-D images of 40 classes, of which 795 are used for training and the remaining 654 are used for testing. The SUN-RGBD dataset has 37 categories and contains 10,335 RGB-D images (5285 for training and 5050 for testing). In the experiment, the data processing includes adjusting resolution, data enhancement, data labeling and data normalization. We adjust all RGB images, Depth images and ground real images to a high spatial resolution of 480 × 640. During training, we use data enhancement to improve data diversity including random scaling, cropping and flipping to the inputs of RGB image and Depth image, respectively. For RGB images, we further enhance them by applying random hue, brightness, and saturation adjustments. The RGB images and Depth images are normalized to 0–1.

During the experiment, we use the same experimental conditions to repeat the experiment for five times to ensure the validity of the evaluation results. The experimental results are the average of five replicates. Through the experiments on NYUDv2 and SUN-RGBD datasets, it can be observed from

Table 2. Comparison with the state-of-the-art models on the NYUDv2 dataset (the top ones are marked in bold).

Method	mIoU (%)	Pixel Acc. (%)
3DGNN [20]	43.1	-
Kong et al. [39]	44.5	72.1
RAFNet [40]	47.5	73.8
ACNet [9]	48.3	-
CANet [41]	51.2	76.6
NANet [36]	51.4	77.1
DCANet [28]	53.3	78.2
MGCNet [24]	54.5	78.7
ConvNeXt-CMFFM	51.9	76.8

and

Table 3. Comparison with the state-of-the-art models on the SUN-RGBD dataset (the top ones are marked in bold).

Method	mIoU (%)	Pixel Acc. (%)
3DGNN [20]	45.9	-
Kong et al. [39]	45.1	80.3
RAFNet [40]	47.2	81.3
ACNet [9]	48.1	-
CANet [41]	48.1	81.6
NANet [36]	48.8	82.3
DCANet [28]	49.6	82.6
MGCNet [24]	51.5	86.5
ConvNeXt-CMFFM	53.5	82.5

that our proposed model achieves good results. For example, the application of our model on the NYUDv2 dataset shows the mIoU of 51.9% (the third out of the nine comparable models) and the pixel accuracy of 76.8% (the fourth out of the nine models). The application of our model on the SUN-RGBD dataset shows the mIoU (53.5%) is the best among the nine comparative models and the pixel accuracy (82.5%) is the third out of the nine models.

Figure 4 shows the segmentation results of our model on NYUDv2 dataset (three indoor scenes including wall, chairs, sofa, clothes and so on in different colors), from which it can be observed that our model can well segment various objects in the scenes with the good segment effect close to the truth.

4.3. Our Rice Dataset

At the same time, we conduct practical application tests on the self-built RICE-RGBD image dataset, which contains two types of data, namely RGB-D data of single rice plant and whole rice cluster, as shown in Figure 5. There are 10,000 RICE-RGBD images in total in the dataset, with 6000 for training and 4000 for testing. Similar to the public dataset, the data processing includes adjusting resolution, data enhancement, data labeling and data normalization. We adjust all RGB images, Depth images and ground real images to a high spatial resolution of 512 × 160. During training, we us data enhancement to improve data diversity including random scaling, cropping and flipping to the inputs of RGB image and Depth image, respectively. For RGB images, we further enhance them by applying random hue, brightness and saturation adjustments. Both the RGB images and Depth images are normalized to 0–1. The ground real images of our rice dataset are labeled. As can be seen from the Figure 5, ConvNeXt-CMFFM is able to well segment the spike and straw of rice. This experiment shows that our model can be well used in practical agricultural application.

4.4. Ablation Study

To further illustrate the validity of our proposed CMFFM, we compared the mIoU and Pixel Acc. metrics of ConvNext with and without the CMFFM on three datasets and the results are shown in

Table 4. Comparison results.

Method	Dataset	mIoU (%)	Pixel Acc. (%)
ConvNeXt-CMFFM	NYUDv2	51.9	76.8
	SUN-RGB	53.5	82.5
	RICE-RGBD	74.8	88.3
ConvNeXt	NYUDv2	50.2	76.1
	SUN-RGB	50.9	79.9
	RICE-RGBD	71.5	86.2

. It shows that ConvNext-CMFFM performs better than ConvNeXt on the three datasets. Specifically, for our rice dataset, compared with ConvNeXt (without CMFFM), the mIoU score of ConvNext-CMFFM is increased from 71.5% to 74.8% and its pixel accuracy is increased from 86.2% to 88.3%. This indicates that CMFFM fusion of data features of different modes can help the model extract more important data features, improve the generalization ability and inference ability of the model, and further promote the segmentation performance of the model.

4.5. Discussion

In our model, the CMFFM proposed by us integrates RGB and Depth image features through channel-wise and spatial-wise phases. Meanwhile, RGB feature and Depth feature complement each other and participate in the calculation of the next layer of their respective branches. The ConvNeXt further enhances CNN feature mining capabilities. It can be seen from the experimental results that the segmentation results of our method on the public datasets of NYUDv2 and SUN-RGBD are competitive. Through the comparison of ablation studies, it can be determined that the proposed CMFFM improves the accuracy of model segmentation, which indicates that our module can realize the deep fusion of RGB features and Depth features. Furthermore, our proposed CMFFM can also be embedded into other backbone networks to improve the segmentation accuracy of models.

In addition, it can be seen that on the dataset of NYUDv2, the segmentation accuracy of our model is lower than that of DCANet and MGCNet, but on SUN-RGBD dataset, the segmentation accuracy of our model is the best with the highest mIoU. It shows that our model has better performance on the dataset with more training samples (for example, 10,335 samples of SUN-RGBD dataset) and slightly poor performance on the dataset with fewer data samples (for example, 1449 samples of NYUDv2). This indicates that although CMFFM proposed by us can integrate the features of RGB and Depth relatively well, it still requires a large number of training samples to learn more critical features, and there is still great room for improvement in the feature fusion of RGB and Depth.

At present, the parameters of our model are $309\times {10}^6$ and FLOPs are $408\times {10}^9$, which can meet the general desktop requirements. The processing speed of the model for a group of data (including one RGB image and one Depth image) is about 0.03 s, that is, the processing speed is about 33 frames per second, which can meet the requirements of common real-time line tasks. However, on embedded devices, the resource requirement of our model makes it difficult to apply it in embedded devices. Reducing the resource overhead of our model will be the future improvement goal.

Thus, the limitations of this model can be summarized as follows. First, the proposed method still needs to be improved for small sample data sets. Second, the two-stream structure makes the model cost a lot of resources, which limits its ability to be transplanted to portable embedded platforms. Finally, our method has a high requirement on the quality of Depth images. When collecting data in actual agricultural scenarios, due to the impact of equipment and environment, the collected depth information may be missing. At this point, this model is not sufficient to fuse the features of RGB and depth.

In future work, we will further explore the correlation between RGB and Depth to achieve a deeper cross-modal feature fusion to employ fewer training samples, learn more key features and reduce the dependence of the model on data. Meanwhile, we are considering to design a lightweight framework to reduce the model’s demand for resources and improve and expand its real-time application performance on embedded system. In addition, in actual agricultural scenarios, we will also consider using RGB image and the complementary Depth information to perform the semantic segmentation of RGB-D.

5. Conclusions

In this paper, we propose a cross-modal feature fusion RGB-D semantic segmentation model based on ConvNeXt to better utilize multi-stage RGB-D features for semantic segmentation. In particular, a novel Cross-modal Feature Fusion Module (CMFFM), embedded in multiple stages of the model, is able to capture both spatial- and channel-wise features in RGB-D features at various stages. Extensive experimental results confirm the effectiveness of our method on the public NYUDv2 and SUN-RGBD datasets. The results on our self-built rice dataset also confirm the practical agricultural application of our method.