Enhancing Autonomous Visual Perception in Challenging Environments: Bilateral Models with Vision Transformer and Multilayer Perceptron for Traversable Area Detection

Abstract

The development of autonomous vehicles has grown significantly recently due to the promise of improving safety and productivity in cities and industries. The scene perception module has benefited from the latest advances in computer vision and deep learning techniques, allowing the creation of more accurate and efficient models. This study develops and evaluates semantic segmentation models based on a bilateral architecture to enhance the detection of traversable areas for autonomous vehicles on unstructured routes, particularly in datasets where the distinction between the traversable area and the surrounding ground is minimal. The proposed hybrid models combine Convolutional Neural Networks (CNNs), Vision Transformer (ViT), and Multilayer Perceptron (MLP) techniques, achieving a balance between precision and computational efficiency. The results demonstrate that these models outperform the base architectures in prediction accuracy, capturing distant details more effectively while maintaining real-time operational capabilities.

1. Introduction

As artificial intelligence has advanced, autonomous vehicles have emerged as a compelling research topic for both industry and academia [1], with various modules being developed for their implementation [2]. One of these modules enables environmental perception so that the vehicle can create an accurate representation of the route and estimate its position in the road ecosystem [3]. Furthermore, a good representation of the environment allows the selection of optimal parameters to save energy during vehicle control, plan routes to reduce traffic congestion, and increase safety during driving [4].

The vehicle during navigation must detect, through embedded perceptual sensors [5], the traversable area and the type of obstacles to avoid. The most commonly implemented perceptual sensors for the operation of autonomous ground vehicles are: RGB camera, LiDAR, and RADAR [6]. RGB cameras are the most popular sensors, even implemented as the only sensor to reconstruct the vehicle’s environment [7] due to their ability to interpret textures, potentially achieve high resolutions, consume low amounts of power, and be more affordable than RADAR or LiDAR [8].

Semantic segmentation consists of assigning a label to each pixel of an input image [9]. Semantic segmentation models based on deep learning have achieved good performance, allowing the description of the environment in complex scenes, the ability to gain information that allows the vehicle to predict the steering angle [10], determine lane boundaries [11], detect obstacles [12], and infer the enabled area for vehicle traffic [13]. Studies have been conducted on structured [14,15] and unstructured [16,17]. The former are routes marked with signage and paved, usually located in urban areas. The latter are unpaved, unmarked routes, which can be found in rural areas and industrial sites [18,19]. Scenes with unstructured routes are challenging for semantic segmentation models based on deep learning because the unmarked route presents little differentiation from the non-traversable area, making it difficult to classify these between classes [20].

Despite significant advances in semantic segmentation for precise detection of the traversable area in unstructured routes, capturing details or high-frequency characteristics in images remains a challenge, especially in environments with a low contrast between the route and the rest of the scene [21,22]. Currently, robust models based on Convolutional Neural Networks (CNNs) have been developed for this purpose, but the most common solution has been to add more parameters to the base architecture [23] or use knowledge distillation techniques [24], which make it difficult to implement the model on devices with limited storage capacities, and it also complicates the training process.

Recent publications show that hybrid architectures based on Convolutional Neural Networks, Vision Transformer (ViT), or Multilayer Perceptron (MLP) capture global characteristics with fewer parameters than pure models, maintaining a high inference capacity [25,26].

Experiments with hybrid models of binary semantic segmentation in low-contrast images show good performance [27,28]. These use CNNs to extract local features and integrate ViT or MLP as an attention module to model global relationships between features extracted by CNNs layers.

Based on the good performance presented by hybrid architectures in low-contrast images and their applications in lightweight systems, the idea of implementing this type of architecture for the detection of vehicular routes in environments where roads are not clearly visible arises. In the present study, ViT and MLP-based skeletons are developed and evaluated in a bilateral model called BiSeNet, combining and evaluating the three techniques: Convolutional Neural Networks, ViT, and MLP, to generate a robust, lightweight model that acts in real-time.

The contributions of the present research are:

• A new architecture based on the bilateral model is developed for the semantic segmentation of unstructured routes to detect the traversable area for a vehicle, achieving greater precision in capturing the details of these scenes compared to the architectures used as backbones. The proposed architectures, which combine CNNs, MLP, and ViT techniques, demonstrate a better ability to capture high frequencies in the images of the databases used.
• A lighter version of the BiSeNet architecture is developed, making it implementable on equipment with limited storage capacity, while maintaining real-time inference. When using the BiSeNet architecture with ViT and MLP as the backbone, it is approximately three times lighter than the BiSeNet version with Xception as the backbone.

The performance of the bilateral BiSeNet architecture is implemented and evaluated considering ViT and MLP-based skeletons on the public CaT and Rellis 3D databases, which contain scenes with low contrast between the traversable area for a standard vehicle and the rest of the ground. The databases were adapted for the training of binary semantic segmentation architectures.

This article consists of seven sections. Section 2 discusses state-of-the-art studies, presenting the main databases created on unstructured routes and models that have been used to study semantic segmentation in route images. The methodology used for the study of the models is described in Section 3, detailing the databases used and the architecture of the studied models. The results obtained and comparative studies are presented in Section 4 and their discussion is carried out in Section 5. Section 6 presents the conclusions of this study and, finally, Section 7 outlines potential future work.

2. State-of-the-Art

2.1. Public Databases of Unstructured Routes

There are many databases that have been published for the study of scene understanding with applications in autonomous vehicles. The vast majority of these databases present 2D annotations for RGB camera images, such as mask annotations for training semantic segmentation models of scenes and object bounding boxes for training object detection models. Most of the existing databases are created on urban roads (on-road), the most used in this category are KITTI [29], CamVid [30], and CityScapes [31].

Databases for the study of vehicle navigation in off-road scenes are scarcer. Generally, these types of databases are created in environments with various types of obstacles, scarce signage, and variable lighting conditions. Some examples of such databases are CaT [32] and AutoMine [33], which focus on capturing industrial environments, such as forestry and mining, respectively. On the other hand, IDD [34], set in an Indian city, is also considered an off-road scene database due to traffic conditions and poor road infrastructure.

2.2. Convolutional Neural Network-Based Models

The first methods for semantic segmentation were developed using CNNs and have led to rapid development in the field of image semantic segmentation. FCN [35] is a pioneer in the encoder–decoder scheme that optimized the segmentation process. Then, UNET [36] improved this scheme by connecting features between the encoder and decoder at different levels, allowing features to remain after the downsampling process. Based on the UNET architecture, BiSeNet [37] was designed, reducing the feature fusion links and implementing a convolutional attention module. This allows reducing the number of parameters and operations, improving the inference time compared to UNET [38].

The first version of BiSeNet consisted of two paths: a contextual one and a spatial one; whereas in its best version, the contextual path is composed of Xception [39] as a backbone and convolutional attention modules. Xception is an architecture that uses depthwise separable convolution [40] in most of its layers. This technique consists of applying depthwise convolution that acts on each channel individually, and then applying pointwise convolution that combines the information from the channels. This approach allows for reducing the number of parameters and computational demands compared to traditional convolutions.

2.3. Vision Transformer-Based Models

The self-attention mechanism of visual transformers allows correlating distant sections and features of the image. The first fully developed architecture using this methodology for semantic segmentation has been ViT [41], which presents good performance, especially in capturing features in global contexts, but has a high computational cost of O(N²), affecting the inference speed. According to [42,43], memory access is the most critical factor affecting the speed of models. Operators, such as frequent reshaping, element-wise addition, and tensor normalization, consume time in accessing memory units. Although there are methods to address this inefficiency with sparse attention [44,45] and low-rank approximation [46,47], they often sacrifice prediction accuracy for inference time.

Recently, methods have been developed that operate in real-time without sacrificing prediction accuracy by reducing memory access time. Swin Transformer [48] divides the image into small windows that are processed locally by the attention modules, reducing cache memory access. The shifted window scheme provides efficiency by limiting the calculation of self-attention to non-overlapping windows while allowing cross-window connection. This architecture presents flexibility to models at various scales with linear computational complexity. LinFormer [49] introduces a linear transformation that reduces the dimensionality of input sequences before calculating attention, reducing the amount of memory needed to store calculated values. The main idea is to add two linear projections when calculating the key and values before applying the attention mechanism. EfficientViT [50] replaces the MHSA module with a CGA module composed of MHSA modules with different channelwise that partition the image features into different heads. The architecture was designed in a modular way, using fewer memory-bound, self-attention layers and memory-efficient FFN layers for communication between channels. After each FFN module, depthwise convolution is used to increase token interaction. FastViT [51] uses structural reparameterization to reduce memory access cost, which reorganizes operations that normally occur within a residual block (skip connections), simplifying operations. On the other hand, overparameterization [52,53] is implemented, which, although increases the number of parameters beyond what is necessary, also facilitates model training and makes it more flexible in inference.

2.4. Multilayer Perceptron-Based Models

With the idea of simplifying deep-learning models, models based only on MLPs [54] and residual connections [55] have recently been created. Compared to CNNs and transformers, these architectures involve less inductive bias and, therefore, are more adaptable to changes during inference [56]. MLP-based models are composed of three parts: patch embedding, channel-mixing, and patch-mixing. The first stage subdivides the image and then passes it to the other stages that relate features and patches [57].

MLP-Mixer [54] was the first proposed model for image classification using only perceptron layers. This model uses multiple perceptron layers to replace the convolution operation of CNNs and the self-attention mechanism of transformers. However, this architecture presents inefficiencies due to its way of relating tokens, that is, mixing different tokens without discriminating that some have more relation between them than others. To optimize the mixing process, variations of this method have been designed. ASMLP [58] reduces the number of relationships in the mixing process through bounded horizontal and vertical shifts. The main block is composed of skip connection, MLP, and axial shift operations, in the latter vertical shift and horizontal shift are used to extract features from the images and channel projection to transform the features using a linear layer. WaveMLP [59] dynamically adds weights to tokens by representing their features as a wave function where the amplitude is the original feature and the phase contains semantic content, allowing the efficient representation of tokens facilitating algorithm access to this information. The model is built by stacking blocks of phase-aware token-mixing, channel-mixing MLPs, and normalization layers in four stages. The gMLP architecture [60] designs gating operations that act as information filters, discriminating between features that should be processed and those that should be ignored. This is achieved through the linear gating operation, which consists of applying element-wise multiplication between the linear function with trainable parameters and the projected channels. S2MLP [61] introduces the spatial-shift module that replaces the channel mixing and token mixing modules of the MLP-MIXER model with a spatial shift that allows local relating of image features. This module contains no trainable parameters, making it a simple and scalable architecture. Before and after applying spatial-shift, fully connected layer blocks are applied that add weight to the relationships between image channels.

2.5. Lightweight Hybrid Models That Operate in Real-Time

Considering that CNNs are fast and efficient at capturing local features and that ViT and MLPs are better at capturing global relationships, lightweight models that combine these techniques have been created, improving the ability to model complex dependencies, capturing fine details, and operating in real-time. The following presents strategies that improve these two aspects using deep learning techniques that often mix CNNs, ViT, or MLP.

Segformer [62] is composed of an MiT-based backbone that applies overlapped patch merging and positional-encoding-free, then applies an efficient attention block. MLP blocks are used to decode the information captured by the transformer architecture. MaxViT [63] reduces the quadratic complexity of typical attention modules to a linear complexity, without losing global feature capture. These modules are interleaved with depthwise CNN modules and squeeze-excitation [64] that optimize local feature capture. MobileViT [65] interleaves MobileNetV2 blocks with a transformer that uses CNNs to optimize the capture of local and global features. The MobileNetV2 blocks apply depthwise separable convolutions to reduce computational complexity, while the transformer inserts global self-attention to capture long-range dependencies.

2.6. Models for Semantic Segmentation of Unstructured Routes

In the design of architectures that operate on unstructured routes, it is important to consider the capture of different textures and their global context in the scene. The following presents strategies that improve these two aspects using deep-learning techniques that often mix CNNs, ViT, or MLP.

In [23], the CMSNet architecture is designed to test different configurations. Resnet or VGG can be used as the backbone. To increase feature capture, various pyramid feature groupings, such as ASPP, GPP, and SPP, were selected. Experiments carried out on a self-created database called Kamino, with a performance of 87% mIoU and an inference speed of 29 FPS. The OFFSEG [66] architecture uses the pretrained MobilNetV2 model on the ImageNet database as a classifier, followed by a color-based segmentation methodology, separating the scene into four classes. The evaluation was carried out on the Rellis3D and RUGD databases, achieving an average accuracy of 97.3%. The study model, based on the encoder–decoder architecture type, proposes and studies models with good performance in one of the two roles. The studies are carried out considering a database generated by the author, where the best performance is presented by the ResNet and DeepLabV3 pair as encoder and decoder, respectively, with an mIoU of 85.2% and an inference speed of 37 FPS, outperforming the performance of the models separately.

The study [67] designs an architecture to capture the traversable area in mining environments, generating its own database. The segmentation network is composed of convolutional layers, transformers, and linear classifiers. Its lightweight version presents a performance of 92.2% mIoU, which exceeds models such as Swin-T and PSPNet.

3. Methodology

The present study followed the methodology presented in Figure 1. It began with a review of the state-of-the-art in public databases and architectures for real-time semantic image segmentation. Then, the databases and the type of base architecture were selected to conduct the study. Finally, the models were trained to evaluate their performance, considering the application where they are implemented.

In this scheme, it is observed that this research begins by taking two routes. On one hand, a review of public databases based on unstructured routes is carried out, and on the other hand, a review of state-of-the-art architectures is conducted. Both routes are linked for the training, validation, and testing of new architectures in challenging unstructured route databases. The arrows graphically indicate the logical sequence in which the present investigation was carried out.

The research stages are detailed below.

3.1. Database Selection and Processing

To evaluate the designed models, public databases created in forests and fields were analyzed. From this group, databases that presented scenes with low contrast between the route and the rest of the ground were selected, with good lighting and weather conditions and without moving obstacles. The annotations had to contemplate masks that delimited the traversable area from the rest of the scene.

None of the analyzed databases present binary annotations; therefore, the original annotations were modified considering the differentiation of the traversable route from the rest of the scene. This process was carried out by grouping the classes of the original annotations into two classes and then binarizing the image.

3.2. Selection and Design of Bilateral Models

Bilateral models for semantic segmentation operate in real-time without sacrificing inference accuracy. These models are composed of two paths, a contextual path and a spatial path [37]. The first path is responsible for increasing the receptive field of the neural network. The second preserves the details of the image that are lost through operations in the contextual path [68]. Additionally, architectures were designed where ViT as a decoder and MLP as an attention module were connected in series in the contextual path.

Due to the function it performs, the contextual path requires more computational resources and execution time than the spatial path. Recent advances in ViT and MLP-based models offer an alternative to improve the capture of global features in the contextual path, reducing the number of parameters of the base architecture without sacrificing real-time operation. After a thorough analysis of architectures, two ViT models and two MLP models were selected to implement in the bilateral model, analyzing the number of parameters, implementation simplicity, and flexibility during inference. Additionally, architectures were designed where ViT was connected in series as a decoder and MLP as an attention module in the contextual path.

3.3. Training of Hybrid Bilateral Models

The databases were divided considering 80% of the images for training, 10% for validation, and 10% for testing the model.

To avoid overfitting, the training and validation data were augmented 10 times by applying 20% variations in contrast, brightness, saturation, and hue. The images were also rotated considering 10° as the angular limit, and the RGB channel values of the image were varied by a maximum of one unit with a 50% probability of occurrence.

During training, the Mean Intersection Over Union (mIoU) performance index was used as the cost function, and the Adam optimization algorithm was used to optimize the parameters. The batch size was 32 images, considering 1000 epochs to train the models and a learning rate of 1 × 10⁻³. The iterations were performed on an NVidia A100 GPU with 80 GB of memory.

3.4. Evaluation of Hybrid Bilateral Models

To measure the accuracy of the segmentation algorithm, the performance indices Mean Intersection Over Union (mIoU) and Dice Similarity Coefficient (DSC) are provided in terms of the number of pixels correctly classified as class i (${\mathrm{T}\mathrm{P}}_{\mathrm{i}}$), incorrectly classified as class i (${\mathrm{F}\mathrm{P}}_{\mathrm{i}}$), and those that were not correctly classified in class i (${\mathrm{F}\mathrm{N}}_{\mathrm{i}}$). M is the total number of classes to be inferred:

$$\mathrm{m}\mathrm{I}\mathrm{o}\mathrm{U}=1/\mathrm{M}\sum _{\mathrm{i}=1}^{\mathrm{M}}{\mathrm{T}\mathrm{P}}_{\mathrm{i}}/({\mathrm{T}\mathrm{P}}_{\mathrm{i}}+{\mathrm{F}\mathrm{P}}_{\mathrm{i}}+{\mathrm{F}\mathrm{N}}_{\mathrm{i}})$$

(1)

$$\mathrm{D}\mathrm{S}\mathrm{C}=1/\mathrm{M}\sum _{\mathrm{i}=1}^{\mathrm{M}}2{\mathrm{T}\mathrm{P}}_{\mathrm{i}}/({2\mathrm{T}\mathrm{P}}_{\mathrm{i}}+{\mathrm{F}\mathrm{P}}_{\mathrm{i}}+{\mathrm{F}\mathrm{N}}_{\mathrm{i}})$$

(2)

It is expected that the predictions of the studied models predict two classes: the traversable route and the rest of the scene. Therefore, M would be equal to two.

The images used to test the designed models were augmented 10 times by applying variations of 20% in contrast, brightness, saturation, and hue. The images were also rotated considering 10° as the angular limit, and the RGB channel values of the image were varied by a maximum of one unit with a 50% probability of occurrence. This was performed to increase the test sample and thus obtain representative statistical values that allow comparing the performance of the models in the selected databases.

Other important parameters for evaluating model performance are inference speed, model parameter count, and trained model computational size [69,70].

The total test images were used to calculate the inference speed, and then the total time was divided by the number of images. The models are evaluated on a computer with two NVidia T4 GPUs with 16 GB of memory each.

4. Results

4.1. Unstructured Route Databases

Table 1. Public databases created on unstructured routes with annotations for the study of models dedicated to the semantic segmentation of images.

Dataset	Images	Resolution	Classes	Scene
RUGD [71]	2700	688 × 550	6	Countryside
Rellis 3D [72]	6235	1920 × 1200	20	Countryside
ORFD [73]	12,198	1280 × 720	3	Countryside and Forest
CaT [32]	1812	1024 × 644	4	Forest

presents public databases for the study of vehicle navigation on unstructured routes that were analyzed to test the models. RUGD [71] contains RGB images with multiple types of annotations and in various types of terrain in rural areas. Rellis 3D [72] is a multimodal database whose data were captured at Rellis Campus of Texas University and which presents the largest number of classes of the reviewed databases. ORFD [73] was built on various unstructured terrains with different weather and lighting conditions. CaT [32] was made in a forest near Mississippi State University and presents annotations to evaluate the traversability of off-road and on-road vehicles.

CaT [32] and Rellis3D [72] databases were selected to train and evaluate the models. Most of their images present a low contrast between the area enabled for vehicular traffic and the rest of the ground. Figure 2 shows examples of the analyzed images.

CaT considers annotations for the traversability of three types of vehicles. For this study, areas that were traversable by smaller vehicles were considered as the area to be segmented, and areas traversable only by 4 × 4 vehicles were excluded.

Rellis 3D presents annotations considering 20 classes, so the mask was processed to contain only two labels: the route enabled for transit and the rest of the scene. As classes/labels enabled for vehicular traffic, the classes with the following annotations were selected: grass, dirt, concrete, mud, and asphalt.

Figure 3 shows the original masks of the CaT and Rellis 3D annotations, as well as the binarized masks prepared for training the designed models.

4.2. Bilateral Architecture Schemes

In Figure 4, the three bilateral schemes studied can be observed. In the BiSeNet architecture, Figure 4a, the contextual path is composed of a backbone followed by an attention module, both composed of CNNs. Figure 4b presents a hybrid model where the contextual path was modified by a ViT or MLP backbone. Based on the studies [27,74], the contextual path of the bilateral architecture presented in Figure 4c is composed of a ViT skeleton and the mixing module of the studied MLP models.

The state-of-the-art ViT-based models were analyzed, concluding that EfficientViT [50] and FastViT [51] offer a better balance between computational efficiency and performance, highlighting their optimization for memory access and inference accuracy. The architecture of both models is presented in Figure 5.

EfficientViT applies spatial mixing between FFN layers to speed up communication between different features, and CGA, which is composed of multiple heads that distribute the total features to process them. FastViT implements RepMixer, a convolutional module that mixes tokens by reparameterizing the residual connections, to alleviate computational memory access. Additionally, dense convolutions located in the stem and patch embedding modules are replaced by a factorized version that uses train-time overparametrization.

Regarding the MLP-based models, the gMLP [60] and S2MLP [61] architectures were selected, which presented a better balance between scalability and prediction efficiency. The gMLP model uses a Spatial Gating Unit module in the token mixing section that makes the feature selection more flexible, allowing adaptation to different data types and tasks. By functioning as a feature filter, the Spatial Gating Unit allows acceleration of the prediction thanks to computational savings and improving accuracy since its mechanism builds nonlinear relationships between the image features. S2MLP implements a mixing module that does not require parameter training, reducing computational time during training. The spatial shift technique, applied in the Spatial Shift module, allows capturing local dependencies in images, which improves the representation capacity without the need to apply convolutional techniques. Figure 6 presents the mixing modules of the aforementioned architectures.

The first version of BiSeNet [37] was selected as the base model because, although the other versions have better performance, they require more memory space and consider a knowledge distillation process that requires greater computational capacity during training [75], and the operating model is less flexible to changes during inference [76].

Table 2. Output tensors of the modules that make up the studied architectures. The tensor format is: [channel, width, height].

Architecture Module	Output Tensor
Input	[3, 256, 256]
Spatial path	[256, 32, 32]
Contextual path	[768, 32, 32]
Features fusion	[2, 256, 256]
Output	[1, 256, 256]

presents the output tensors of the modules of the bilateral architectures when inputting 256 × 256 and three RGB channel images. The images are passed consecutively through the contextual and spatial paths, reducing their resolution and increasing the number of channels. In the feature fusion module, the resolution is re-established with two channels where each of the detected classes is specified.

4.3. Training of Bilateral Hybrid Models

Considering the parameters for training presented in the methodology, the curves shown in Figure 7 were obtained for the bilateral models with the best performance.

In terms of training speed, the bilateral models exhibit higher rates than the models used as backbones. While BiSeNet-EfficientViT-S2MLP achieves a training rate of 28.71 it/s, EfficientViT has a training rate of 7.76 it/s. Similarly, BiSeNet-FastViT-S2MLP and its backbone FastViT exhibit training rates of 28.71 it/s and 2.76 it/s, respectively. BiSeNet-Xception achieves a training rate of 22.92 it/s.

4.4. Evaluation of Bilateral Hybrid Models

Table 3. Performance indices of the investigated models considering the CaT and Rellis 3D datasets.

Model	Mean mIoU CaT (%)	Mean mIoU Rellis 3D (%)	Mean DSC CaT (%)	Mean DSC Rellis 3D (%)	Speed (FPS)	Parameters (M)	Trained Model Weight (MB)
BiSeNet-Xception	92.75	96.99	96.03	98.43	98.37	22.42	90.07
BiSeNet-gMLP	82.31	95.95	89.36	97.8	271.26	3.35	324.87
BiSeNet-S2MLP	83.86	96.43	90.69	98.10	73.98	68.37	324.87
FastViT	93.01	96.28	96.16	98.06	94.53	3.16	13.32
BiSeNet-FastViT	93.17	96.57	96.22	98.17	77.06	3.64	16.24
BiSeNet-FastViT-gMLP	93.15	96.77	96.22	98.31	78.05	5.09	22.04
BiSeNet-FastViT-S2MLP	93.51	96.97	96.42	98.46	74.17	7.18	30.44
EfficientViT	91.81	96.17	95.36	98.02	127.04	3.41	13.76
BiSeNet-EfficientViT	91.57	96.41	95.2	98.10	96.46	3.89	63.93
BiSeNet-EfficientViT-gMLP	92.34	96.66	95.66	98.26	92.2	5.34	22.87
BiSeNet-EfficientViT-S2MLP	92.28	96.67	95.68	98.26	79.24	7.44	31.27

presents the performance metrics and key parameters of the base and designed architectures, considering the two studied datasets. It is observed that BiSeNet-FastViT-S2MLP outperforms the base architectures in terms of prediction accuracy. When using the architecture with gMLP, while detail capture is observed, it is less than with the S2MLP implementation. This table can be used for an ablation study of the designed hybrid models where a ViT model was used as a backbone and an MLP model as the attention module.

Figure 8 presents boxplots to analyze the distribution of the mIoU index results, considering both datasets. To facilitate the analysis,

Table 4. Percentile, median, maximum, and minimum values used to create the boxplots in Figure 8.

Architecture	BiSeNet-EfficientViT-S2MLP		BiSeNet-FastViT-S2MLP		BiSeNeT-Xception
Dataset	CaT	Rellis3D	CaT	Rellis3D	CaT	Rellis3D
Q1 (25%)	0.875	0.951	0.886	0.953	0.884	0.957
Mediana	0.940	0.968	0.952	0.970	0.943	0.972
Q3 (75%)	0.973	0.950	0.984	0.981	0.974	0.98
Minimum	0.728	0.906	0.739	0.911	0.749	0.918
Maximum	0.999	0.994	0.999	0.994	0.999	0.994

shows their main parameters.

Figure 9 and Figure 10 show segmentation examples of paths using the designed architectures and ViT models, considering the CaT and Rellis 3D datasets, respectively. The red circles with dashed lines emphasize areas with high intensity frequencies, which are challenging for the cited models to capture. The images presented as examples were selected to be representative of each segment traversed by the vehicle.

The graph in Figure 11 allows comparing the inference speed and accuracy in predicting the traversable area of state-of-the-art hybrid architectures with the best proposed design, BiSeNet-FastViT-S2MLP, and the CaT dataset. The hybrid models from the state-of-the-art architectures were selected so that the computational weight of the trained model is around 30 MB. These models are available in the Timm library of PyTorch.

5. Discussion

In the model design, the goal is to enhance the function of the contextual path by testing two backbones based on ViTs and MLPs. The ViT models focus on reducing memory usage through various modules, while the MLP models focus on the complex classification of semantic information.

The hybrid models have a similar number of parameters, approximately one-third of Xception’s. However, the training speeds of both models are practically identical, despite the backbones differing significantly in their training rates. This suggests that the bilateral base architecture influences training speed more than the choice of backbone. When these architectures are integrated into BiSeNet, the data flow is split between the contextual and spatial paths, alleviating part of the computational burden previously managed entirely by the backbone. Additionally, the use of skip connections improves model convergence without causing gradient problems or extending backpropagation operations.

The training curves of the best-performing bilateral models show better convergence of the BiSeNet-EfficientViT-S2MLP model, comparable to BiSeNet-Xception. However, BiSeNet-FastViT-S2MLP shows greater convergence in its validation curve, indicating that the model finds the correct inference parameters more efficiently.

Both selected datasets come from forest environments, where images have low contrast between the path and the surrounding ground. Particularly, the scenes in the CaT dataset pose more challenges for the models due to the abundance of vegetation, causing the entire scene to have a similar intensity, complicating the path distinction. This challenge is reflected in both the quantitative and qualitative results. The boxplots reveal that the inference results for the CaT dataset are more dispersed, presenting multiple outliers, while the results for Rellis 3D are close to the respective boxplot’s minimum and maximum. Despite this, the medians of the bilateral models, considering both datasets, are similar in magnitude.

In terms of performance, the BiSeNet-FastViT-S2MLP model outperforms the base architectures in prediction accuracy. While the gMLP variant shows some capability to capture details, these are fewer compared to the S2MLP implementation.

When comparing the designed model with the base ViT models, a reduction in inference speed and an increase in model weight are observed. This occurs due to the addition of modules to the base transformer architecture, namely the spatial path and the MLP-mixer module. Despite this, the model still operates in real time (over 30 FPS). Although these parameters are affected, accuracy is improved, which is significant for defining contours, and is crucial for an autonomous vehicle to predict steering angles accurately or detect obstacles at a greater distance.

The results sequentially demonstrate the evolution of the traversable path detection as a spatial path and MLP-Mixer modules are added. Both FastViT and EfficientViT show improvements in shape detection when these modules are included. The qualitative results obtained with BiSeNet-FastViT-S2MLP are similar to those obtained with BiSeNet-Xception. Still, it is important to note that the designed architecture has fewer trainable parameters and lower computational weight, which is an advantage in applications on portable devices with limited computational storage.

When comparing BiSeNet-FastViT-S2MLP with state-of-the-art other hybrid models, the proposed model achieves the highest prediction accuracy while maintaining high inference speed. The analyzed models commonly integrate combinations of CNNs-ViT or CNNs-MLP, suggesting that the combined use of the three deep learning techniques—CNNs, ViT, and MLP—as in this study, could further enhance the model’s inference capabilities.

The quantitative and qualitative results obtained for two datasets with similar characteristics demonstrate that the BiSeNet-FastViT-S2MLP architecture not only surpasses the architectures used as a base but also maintains an inference speed compatible with real-time applications. This achievement is particularly significant, considering the complexity inherent in the analyzed datasets, suggesting that the proposed approach could have a substantial impact on improving the visual perception of autonomous vehicles in challenging environments.

6. Conclusions

This study presents a significant contribution to the field of visual perception for autonomous vehicles, demonstrating that it is possible to improve accuracy in detecting traversable areas on unstructured roads without compromising computational efficiency. The development of bilateral hybrid architectures combining the three most well-known techniques in computer vision and deep learning is presented. Among the designed models, BiSeNet-FastViT-S2MLP stands out, showing better low-frequency feature capture. Additionally, when comparing the performance of BiSeNet-FastViT-S2MLP with other hybrid models on the CaT dataset, the model demonstrates better inference accuracy while maintaining real-time operation.

7. Future Works

The models were only tested on isolated images, which does not reflect the reality of practical applications where image sequences are handled. This limitation arises due to the lack of unstructured road datasets that include semantic segmentation annotations in videos, preventing more representative tests. There are also no datasets that contemplate failures in camera capture due to lens fogging, glare from a front light beam, or mobile vibrations.

Hybrid models open new opportunities for developing more robust and adaptable perception systems in various environments. In adverse weather conditions, perception models that extract features from images may generate erroneous predictions [77]. To mitigate this issue, future work could focus on training models on a newly created database containing images with such conditions, allowing the receptive field of hybrid models to improve performance in these scenarios. Another viable solution would be the development of a multimodal platform that combines additional perceptual sensors, such as LIDAR or radar, along with RGB cameras. This would provide complementary data to enhance model predictions and generate more accurate inferences under complex conditions.

This work could serve as a foundation for developing multitasking models that can be implemented on real platforms, such as (Advances Driver Assistance Systems) ADAS. These systems warn drivers about permitted traffic zones and obstacle detection on the road. Additionally, ADAS could be complemented with Autonomous Visualization Agents (AVA) systems [78], which would enhance the driver’s visual analysis by generating dynamic visualizations. AVA, using natural language instructions instead of advanced computer vision tools, would facilitate visual interpretation and decision-making during driving.