MonoGhost: Lightweight Monocular GhostNet 3D Object Properties Estimation for Autonomous Driving

Abstract

Effective environmental perception is critical for autonomous driving; thus, the perception system requires collecting 3D information of the surrounding objects, such as their dimensions, locations, and orientation in space. Recently, deep learning has been widely used in perception systems that convert image features from a camera into semantic information. This paper presents the MonoGhost network, a lightweight Monocular GhostNet deep learning technique for full 3D object properties estimation from a single frame monocular image. Unlike other techniques, the proposed MonoGhost network first estimates relatively reliable 3D object properties depending on efficient feature extractor. The proposed MonoGhost network estimates the orientation of the 3D object as well as the 3D dimensions of that object, resulting in reasonably small errors in the dimensions estimations versus other networks. These estimations, combined with the translation projection constraints imposed by the 2D detection coordinates, allow for the prediction of a robust and dependable Bird’s Eye View bounding box. The experimental outcomes prove that the proposed MonoGhost network performs better than other state-of-the-art networks in the Bird’s Eye View of the KITTI dataset benchmark by scoring 16.73% on the moderate class and 15.01% on the hard class while preserving real-time requirements.

1. Introduction

The development of driving assistance systems holds the possibility of reducing accidents, reducing environmental emissions, and easing the stress associated with driving [1,2,3,4]. Several levels of automation have been proposed, based on their technology capacities and human interaction [5,6]. The most widely known levels can be broken down into six groups [7]. Beginning with Level 0 (Driver-Only Level), the complete control of the vehicle, including steering, braking, accelerating, and decelerating, is completely under the control of the driver [8]. As the level increases from Level 1 to Level 3, the user interaction is reduced and the level of automation is increased [8]. In contrast to earlier levels of autonomous driving, Level 4 (High Driving Automation) and Level 5 (Full Driving Automation) attain fully autonomous driving, where the vehicle can be operated without the need for any driving experience or even a driving licence [9]. The difference between Level 4 and Level 5 is that autonomous vehicles categorized in Level 5 can drive entirely automatically in all driving domains and require no human input or interaction. Thus, Level 5 prototypes remove the steering wheel and the pedals; hence, the role of the driver is diminished to that of a mere passenger [10]. Consequently, there is a constraint on any vehicle equipped with a driving assistance system in order to evolve into a practical reality. This vehicle must be equipped with a perception system that enables high levels of awareness and intelligence necessary to deal with stressful real-world situations, make wise choices, and always act in a manner that is secure and accountable [5,11,12].

The perception system of autonomous cars performs a variety of tasks. The ultimate purpose of these tasks is to fully understand the vehicle’s immediate surroundings with low latency so that the vehicle can make decisions or interact with the real world. Object detection, tracking, and driving area analysis are examples of such activities. 3D object properties detection is a significant area of study in the autonomous vehicles perception system [13]. Current 3D object detection techniques can be primarily classified into Lidar-based or vision-based techniques [14]. Lidar-based 3D object detection techniques are accurate and efficient but expensive, which limits their use in industry [15]. On the other hand, vision-based techniques [16] can also be categorized into two distinct groups: monocular and binocular vision. Vision-based perception systems are widely used due to their low cost and rich features (critical semantic elements within the image that carry valuable information and are used to understand and process the visual data). The most significant downside of monocular vision is that it cannot determine depth directly from image information, which may cause errors in 3D pose estimation in monocular object detection. The cost of binocular vision is higher than that of monocular vision, although it can provide more accurate depth information than monocular vision. Moreover, binocular vision yields a narrower visual field range, which cannot meet the requirements of certain operating conditions [17].

For the case of monocular vision systems, the camera projects a 3D point (defined in the 3D world’s coordinate frame of the object) into the 2D image coordinate frame. This is a forward mathematical operation, which removes the projected object’s depth information. Thus, the inverse projection of the point from the 2D image coordinate frame back into the 3D world’s coordinate frame of the object is not a trivial mathematical task [18].

A 2D object detector block is fed with the 2D captured image and outputs the 2D coordinates of the object of interest defined in the 2D image coordinate frame as well as a cropped image of that particular object (inside the 2D bounding box). The target of the 3D perception system is to recover the object’s 3D bounding box described in the 3D world’s coordinate frame as well as its Bird’s Eye View. The Bird’s Eye View posture is predicted by neglecting the height of the object. A plethora of algorithms have been developed in the literature to perform this task. These algorithms will be discussed in the following section.

In this study, the proposed MonoGhost network, which is a lightweight deep learning-based technique, is proposed to estimate the object’s 3D posture (Position and Orientation) and 3D dimensions from the monocular observed 2D bounding box (a rectangular or square-shaped region that encapsulates and outlines a specific object or region of interest within an image). The estimated object’s orientation and 3D dimensions are then used to generate the Bird’s Eye View of the detected object. The proposed MonoGhost network can use the state-of-the-art 2D object detectors [19,20,21], which can be enhanced to 3D object detectors by training an efficient lightweight feature extractor. Its extracted features are then used to predict the object’s 3D bounding box orientation and dimensions. Taking into account the estimated 3D bounding box orientation and dimensions as well as the coordinates of the object’s 2D bounding box, the object’s Bird’s Eye View bounding box can be generated. This paper is restricted to addressing only objects of class “Car”.

The related work regarding the main methodologies of the 3D object detection are discussed in Section 2. The proposed MonoGhost network, which is explained in Section 3, provides the following contributions:

• A deep learning approach for estimating the Bird’s Eye View bounding box of the detected object, depending on encoding the object’s geometric and visual features using 1D convolution then fusing the encoded features to decode the object Bird’s Eye View translation center.
• Preserving the stability of object’s depth prediction for KITTI [22] hard object case without sacrificing the orientation prediction accuracy.
• Simple design of MonoGhost network, which is composed of low computational burden operations.
• Selection of an efficient lightweight, embedded-device-friendly feature extractor.

The experimental work of the proposed MonoGhost network, which explains the implementation details, the utilized dataset for training and benchmarking, and the training procedures, is discussed in Section 4. The success of the proposed network architecture is verified by experimental results on KITTI Bird’s Eye View benchmark (https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=bev, accessed on 5 November 2023). It is compared to the most effective monocular 3D object identification methods currently available like PGD-FCOS3D [23], KM3D-Net [24], and SMOKE [25], as addressed in Section 5. Conclusions are presented in the final section of the paper.

2. Related Work

The various methods employed for 3D object detection can be classified into two distinct categories: conventional techniques or deep learning techniques. This section provides a comprehensive summary of the present 3D object detection methodologies as summarized in Figure 1. The 3D-data-based techniques in both categories achieve superior detection results over 2D-data-based techniques. However, they come with additional cost related to the sensor setup, as mentioned earlier. On the other hand, 2D-data-based techniques are cost efficient, which comes with sacrificing the depth estimation accuracy [15,17].

2.1. Conventional Techniques

Conventional techniques depend heavily on a prior knowledge of the object appearance to retrieve the essential suite of object features using hand-crafted feature descriptors, followed by the formation of a database by appropriately attributing features with the 3D models [26,27,28]. In order to provide quick detection results, the model database is organized and searched using effective indexing algorithms such as hierarchical RANSAC search, and voting is performed by using ranked criteria [29]. Other approaches depend on different hashing techniques like geometric hashing [26] and elastic hash table [30]. The conventional techniques can be categorized according to the input information into 3D-data-based techniques and 2D-data-based techniques [31].

2.1.1. 3D-Data-Based Techniques

With the advancement of sensor technology, more devices capable of capturing 3D environmental data, such as depth cameras and 3D scanners [32,33], are developed. When compared to 2D-data techniques, 3D-data techniques retain the object’s genuine physical characteristics, which make them a superior choice to quantify the 6D pose [34]. The two primary categories of conventional 3D-data-based approaches are as follows.

Local-Descriptor-Based Approaches: These depend on the local descriptor and utilize an offline global descriptor applied to the model. This global descriptor needs to be rotation and translation invariant. Then, the local descriptor is subsequently generated online and matched with the global descriptor. Super Key 4-Points Congruent Sets (SK-4PCS) [35] can be paired with invariant local properties of 3D shapes to optimize the quantity of data handled. The iterative Closest Point (ICP) [36] approach is a traditional method that can compute the pose transformation between two sets of point clouds. Another approach [37] establishes a correspondence between the online local descriptor and the saved model database by employing the enhanced Oriented Fast and Rotated BRIEF (ORB) [38] feature and the RBRIEF descriptor [39]. A different approach centered around semi-global descriptors [40] can be used to assess the 6D pose of large-scale occluded objects.

Matching-Based Approaches: The objective of matching-based approaches is to find the template in the database that is almost identical to the input sample as well as retrieving its 6D pose. An innovative Multi-Task Template Matching (MTTM) framework was proposed in [41] to increase matching reliability. It locates the closest template of a target object while estimating the masks of segmentation and the object’s pose transformation. In order to enhance the data storage footprint and the lookup searching time, fused Balanced Pose Tree (BPT) and PCOF-MOD (multimodal PCOF) under optimum storage space restructuring was proposed in [42] in order to yield memory-efficient 6D pose estimation. A study on the control of micro-electro-mechanical system (MEMS) microassembly was conducted in [43] to allow for an accurate control to enable minimizing 3D residuals.

2.1.2. 2D-Data-Based Techniques

To measure the 6D pose of objects, a variety of features that are represented in the 2D image can be used. These features may include texture features, geometric features, color features, and more [31,44,45]. Speeded Up Robust Features (SURF) [46] and Scale-Invariant Feature Transform (SIFT) features [47] are considered the primary features that can be utilized for object pose estimation. Color features [48], as well as geometric features [49], can be utilized to enhance the pose estimation performance. The conventional 2D-data-based techniques can be further broken down according to the used matching template into CAD-model-based approaches and real-appearance-based approaches.

CAD-Model-Based Approaches: CAD-modelbbased approaches depend mainly on the rendered templates of CAD models. These approaches are suitable for industrial applications, because illumination and blurring have no effect on the rendering process [31]. A Perspective Cumulated Orientation Feature (PCOF) based on orientation histograms was proposed in [50] to estimate a robust object’s pose. The Fine Pose parts-based Model (FPM) [51] was implemented to localize objects in a 2D image using the given CAD models. Moreover, the edge correspondences can be used to estimate the pose [52]. The multi-view geometry can be adapted in order to extract the object’s geometric features [53], while the epipolar geometry can be used to generate the transformation matrix [54]. Some other approaches suggest visuals-based tracking like [55], which performs tracking approach based on CAD model for micro-assembly and like [56] which employs 6D posture estimation for end-effector tracking in a scanning electron microscope to enable higher-quality automated processes and accurate measurements.

Real-Appearance-Based Approaches: Despite the accuracy that can be achieved by employing 3D CAD models, reliable 3D CAD models are not always obtained. Under these conditions, real object images are used as the template. The 6D pose of an object can be measured depending on multi-cooperative logos [57]. The Histogram Of Gradients (HOG) [58] is an effective technique for improving the pose estimation performance. Another approach [59] suggested template matching in order to provide stability against small image variations. As obtaining sufficient real image templates is laborious and time-consuming, and producing images via CAD model is becoming simpler, approaches that depend on CAD models are much more popular rather than approaches that depend on real appearance [31].

The main downside of the conventional techniques is that they are very sensitive to noise and are not efficient in terms of computational burden and storage resources [60]. The performance of the 2D-data-based techniques heavily depends on the total number of templates, which impacts the pose estimation accuracy, as these approaches depend on the object appearance experience. The greater the number of templates, the more accurate the posture assessment [61,62]. Consequently, a large number of templates necessitates a significant amount of storage and search time [63]. Also, it is not practical to obtain 360° features of the model [64,65]. Because of how the 3D-data-based approaches function, the major limitation for these approaches is that they fail to function adequately whenever the object has a high level of reflected brightness [31]. Another problem is that the efficiency of these approaches is somewhat limited, as point clouds and depth images involve an enormous quantity of data, leading to a large computational burden [31].

2.2. Deep Learning-Based Techniques

Convolution Neural Networks (CNNs) are commonly used in 3D object detection methodologies based on deep learning strategies to retrieve a hierarchical suite of abstracted features from each object in order to record the object’s essential information [66]. Unlike conventional 3D object detection techniques, which depend on hand-crafted feature descriptors, deep learning-based techniques utilize learnable weights that can be automatically tuned during the training phase [67]. Thus, these techniques provide more robustness against environmental variations. The deep learning techniques can also be categorized into 3D- and 2D-data-based techniques, as shown in Figure 1.

2.2.1. 3D-Data-Based Techniques

In recent years, LiDAR-based 3D detection [68,69,70,71] has advanced rapidly. LiDAR sensors collect accurate 3D measurement data gathered from the surroundings in the pattern of 3D points (x, y, z), at which x, y, z are each point’s absolute 3D coordinates. LiDAR point cloud representations, by definition, necessitate an architecture that allows convolution procedures to be performed efficiently. Thus, deep learning-based techniques, which depend on LiDAR 3D detection, can be divided into Voxel-based approaches, Point-based approaches, Pillar-based approaches, and Frustum-based approaches [72,73].

Voxel-Based Approaches: These partition point clouds into similarly sized 3D voxels. Following that, for every single voxel, feature extraction can be used to acquire features from a group of points. The aforementioned approach minimizes the overall size of the point cloud, conserving storage space [73]. Voxel-based approaches, such as VoxelNet [74] and SECOND [75], augment the 2D image characterization into 3D space by splitting the 3D space into voxels. Other deep learning models like Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection [76], Afdet [77], Center-based 3D object detection and tracking [78], and Psanet [79] utilize the Voxel-based approach to perform 3D object detection.

Point-Based Approaches: These were introduced to deal with raw unstructured point clouds. Point-based approaches, such as PointNet [80] and PointNet++ [81] issue raw point clouds as input and retrieve point-wise characteristics for 3D object detection using formations such as multi-layer perceptrons. Other models like PointRCNN [71], PointRGCN [82], RoarNet [83], LaserNet [84], and PointPaiting [85] are examples of deep learning neural networks that adopt a Point-Based approach.

Pillar-Based Approaches: These structure the LiDAR 3D point cloud into vertical columns termed as pillars. Utilizing pillar-based approaches enables the tuning of the 3D point cloud arranging process in the x–y plane, removing the z coordinate, as demonstrated in PointPillars [86].

Frustum-Based Approaches: These partition the LiDAR 3D point clouds into frustums. The 3D object detector models that crop point cloud regions recognized by an RGB image object detector include SIFRNet [87], Frustum ConvNet [88], and Frustum PointNet [89].

2.2.2. 2D-Data-Based Techniques

RGB images are the main input for deep learning-based techniques that rely on 2D data. Despite the outstanding performance of 2D object detection networks, generating 3D bounding boxes merely from the 2D image plane is a substantially more complicated challenge owing to the lack of absolute depth information [90]. Thus, the 2D-data-based techniques for 3D object detection can be categorized according to the adopted method to generate the depth information into Stereographically Based Approaches, Depth-Aided Approaches, and Single-Image-Based Approaches.

Stereographically Based Approaches: Strategies in [91,92,93,94,95] process the pair of stereo images using a Siamese network and create a 3D cost volume to determine the matching cost for stereo matching using neural networks. The preliminary work 3DOP [96] creates 3D propositions by tinkering with a wide range of extracted features such as stereo reconstruction, and object size previous convictions. MVS-Machine [97] considers differentiable projection and reprojection for improved 3D volume construction handling from multi-view images.

Depth-Aided Approaches: Due to the missing depth information in single monocular image input, several researchers tried to make use of the progress in depth estimation neural networks. Previous research works [98,99,100] convert images into pseudo-LiDAR perceptions by harnessing off-the-shelf depth map predictors and calibration parameters. Then, they deploy established LiDAR-based 3D detection methods to output 3D bounding boxes, leading to lesser effectiveness. D4LCN [101] and DDMP-3D [102] emphasize a fusion-based strategy between image and estimated depth using cleverly engineered deep CNNs. However, most of the aforementioned methods that utilize off-the-shelf depth estimators directly pay substantial computational costs and achieve only limited improvement due to inaccuracy in the estimated depth map [103].

Single-Image-Based Approaches: Recently, several research works [13,104,105,106,107] employed only a monocular RGB image as input to 3D object detection. PGD-FCOS3D [23] establishes geometric correlation graphs between detected objects then utilizes the constructed graphs to improve the accuracy of the depth estimation. Some research works like RTM3D [108], SMOKE [25], and KM3D-Net [24] anticipate key points of the 3D bounding box as an adjacent procedure for establishing spatial information of the observed object. Decoupled-3D [109] presents an innovative framework for the decomposition of the detection problem into two tasks: a structured polygon prediction task and a depth recovery task. QD-3DT [110] presents a system for tracking moving objects over time and estimating their full 3D bounding box from an ongoing series of 2D monocular images. The association stage of the tracked objects depends on quasi-dense similarity learning to identify objects of interest in different positions and views based on visual features. MonoCInIS [111] presents a method that utilizes instance segmentation in order to estimate the object’s pose. The proposed method is camera-independent in order to account for variable camera perspectives. MonoPair [112] investigates spatial pair-wise interactions among objects to enhance detection capability. Many recent research works depend on a prior 2D object detection stage. Deep3Dbox [113] suggests an innovative way to predict orientation and dimensions. M3D-RPN [114] considers a depth-aware convolution to anticipate 3D objects and produces 3D object properties with 2D detection requirements. GS3D [115] extends Deep3Dbox with a feature extraction module for visible surfaces. Due to the total loss of depth information and the necessity for a vast search space, it is not trivial to estimate the object’s spatial position immediately [17]. As a result, PoseCNN [116] recognizes an object’s position in a 2D image while simultaneously predicting its depth to determine its 3D position. Because rotation space is nonlinear, it is challenging to estimate 3D rotation directly with PoseCNN [17].

Although deep learning methods that rely on a single RGB image provide some appropriate projection-based constraints, they fail to achieve promising results due to an absence of depth spatial data [98,117].

3. MonoGhost Network

The proposed MonoGhost network can be categorized as a deep learning single image approach based on prior 2D bounding box detection. The mentioned approaches in the previous section that belong to the same category rely on a prior 2D detection stage. They fail to achieve promising depth results depending only on the projection constraint. Moreover, it is not easy to output a reliable depth estimation while maintaining good orientation performance [17].

The proposed MonoGhost relies on a features-fusion architecture to satisfy the projection constraint and output a stable object depth included in the object’s Bird’s Eye View. The proposed MonoGhost network, as will be presented in the remaining part of the section, can output steady object depth without sacrificing good orientation estimation performance while preserving real-time performance.

By relying on the assumption that a 3D bounding box’s perspective projection should closely fit within its 2D observed bounding box, it is possible to build upon the accomplishments of previous efforts on 2D object detection for 3D bounding box estimation [118]. This is why MonoGhost was designed with the ability to be seamlessly integrated with any state-of-the-art 2D object detector. MonoGhost relies on Faster R-CNN [119] to provide the 2D bounding box coordinates for all the results obtained in this research.

As Figure 2 shows the angles, dimensions, and translations on an object model, the 3D bounding box could be described by its dimensions D $=[d_x,d_y,d_z]$, center coordinates T $={[t_x,t_y,t_z]}^T$, and orientation R $(\theta ,\varphi ,\alpha )$, which are characterized by azimuth ($\theta$), elevation ($\varphi$), and roll ($\alpha$) angles. For the assumed case of a flat ground, $\varphi$ and $\alpha$ angles are safely considered zero. Moreover, by considering all the objects to be on the ground, it is also valid to fix the height of the object ($t_y$) to zero. Thus, it is sufficient to predict the Bird’s Eye View posture.

It is then required to calculate the projection of a 3D point $X_o={[X,Y,Z,1]}^T$, at which X, Y, Z are the point’s 3D coordinates described in the world’s 3D coordinate frame, which originates at the object’s center T, into the 2D image coordinate frame, $x={[u,v,1]}^T$. This can be performed, as in [18,113,120], and as detailed in Equation (1), given the object’s posture in the camera coordinate frame $(R,T)\in SE\left(3\right)$ and the intrinsic parameters of the camera, which are represented by matrix K:

$$x=K\left[RT\right]X_o$$

(1)

Therefore, in order to recover the 3D coordinates of the object from 2D image, full representation of the object’s geometric features (Dimensions and Orientation) must be estimated first. That is why the proposed MonoGhost network, shown in Figure 3, first estimates the orientation and dimensions of the object (through the Orientation-Dimensions Estimator block). These estimates are then joined with the object’s 2D bounding box coordinates and the Guiding Depth, which represents the initial depth estimation of the object, to generate the required 3D Bird’s Eye View posture (through the Bird’s Eye View Center Estimator block). The proposed model learns to transform 2D detection from the front camera view into a Bird’s Eye View occupancy map. Consequently, the proposed MonoGhost network is composed of two main stages as presented in Figure 3:

• First stage (Orientation-Dimensions Estimator) accepts the cropped object image as well as the coordinates of the 2D detected bounding box, then extracts object visual features and outputs the object geometric features (orientation and dimensions).
• Second stage (Bird’s Eye View Center Estimator) fuses the object’s visual and geometric features with the 2D bounding box coordinates and the Guiding Depth to estimate the object Bird’s Eye View bounding box center.

3.1. Orientation Estimation

To estimate the global orientation of the object $R\in SO\left(3\right)$, it is insufficient to depend only on the 2D detection’s contents, since the position of the 2D bounding box in the plane of the image is also unavoidable. Consider the rotation R($\theta$) with only azimuth as a parameter, $\theta$ (yaw).

Figure 4 illustrates an instance of a vehicle crossing the road. The vehicle’s global direction $\theta$ does not alter, but its local direction ${\theta }_l$ which is estimated depending on the appearance of the vehicle in the 2D bounding box is changed. Thus, the global direction $\theta$ is calculated by adding the change in the local direction ${\theta }_l$ with respect to ${\theta }_{ray}$, which represents the ray traversing the center of the cropped bounding box originating from the camera center. Since the direction of a ray at a given pixel can be easily calculated given intrinsic camera parameters, it is crucial to estimate the local orientation ${\theta }_l$ using the 2D bounding box’s extracted features. Then, by adding the estimated local orientation ${\theta }_l$ and the ray direction to the center of the observed 2D bounding box ${\theta }_{ray}$, the object’s global orientation $\theta$ is calculated.

As shown in Figure 5, The Orientation-Dimensions Estimator utilizes the GhostNet [121] as a backbone to extract semantic visual feature of the object. Then the extracted features are shared between two main estimation heads. The first estimation head is the Local Orientation Estimation Head that estimates the object’s local orientation ${\theta }_l$. The estimated ${\theta }_l$ is then added to the calculated ${\theta }_{ray}$ in order to output the object’s global orientation $\theta$. The second estimation head is the Dimensions Estimation Head that outputs the 3D object’s dimensions D $=[d_x,d_y,d_z]$.

Object detectors such as YOLO [19] and SSD [122] partition the space of potential bounding boxes into a number of distinct modes known as anchor boxes, and then quantify the continuous offsets that must be adapted to each anchor box. Following a similar idea, the MultiBin architecture [113] can be used for local orientation estimation. It first discretizes the orientation angle space by dividing it into n overlapping bins. For each bin, the Local Orientation Estimation Head predicts both the score probabilities $scor{e}_i$ that the orientation angle is enclosed within the $i^{th}$ bin and the residual correction angle that must be added to the direction of this bin’s center angle to yield the specified orientation angle. The residual rotation angle is denoted by two numbers: sine and cosine of that angle. As a result, each bin i has three outputs: $scor{e}_i$, $sine(\Delta {\theta }_i)$, and $cosine(\Delta {\theta }_i)$.

Consequently, the MultiBin orientation has a total loss of [113], as detailed in Equation (2):

$$L_{\theta }=L_{score}+\alpha \times L_{residual}$$

(2)

The softmax loss [123] of the scores for each bin determines the score loss $L_{score}$. The residual loss $L_{residual}$ tends to reduce the gap between the predicted angle and the ground truth angle. Therefore, the residual loss $L_{residual}$, is equivalent to maximizing cosine distance and can be calculated as explained in Equation (3) [113]:

$$L_{residual}=\frac{1}{n_{{\theta }^{*}}}\sum cos({\theta }^{*}-c_i-\Delta {\theta }_i)$$

(3)

where $n_{{\theta }^{*}}$ is the number of bins that cover the ground truth angle ${\theta }^{*}$. $c_i$ is the angle of the $i^{th}$ bin center. $\Delta {\theta }_i$ is the adjustment that must be made to the center of bin i.

As shown in Figure 6, in the MonoGhost network the orientation angle space is divided into two bins. Thus, the first branch in the Local Orientation Estimation Head has (1280, 256, 2) Fully Connected Network (FCN) units and outputs the bins scores, while the last branch is composed of (1280, 256, $2\times 2$) FCN units and generates the residual correction angles sine and cosine.

3.2. Dimensions Estimation

Cars, vans, trucks, pedestrians, cyclists, and buses are all separate categories in the KITTI dataset [22]. It is noted easily that the objects in each category have a similar shape and size. As an illustration, the dimensions variance for bicycles and cars have limits of centimeters. As a result, instead of depending on a discrete-continuous loss, such as the MultiBin loss, the L2 loss [124] can be used directly. For each dimension it is convenient to predict the deviation value from the average parameter value calculated over the training dataset. The dimensions estimation loss can be calculated as the explanation provided in Equation (4) [113]:

$$L_{Dimensions}=\frac{1}{n}\sum {(D^{*}-\overline{D}-\delta )}^2$$

(4)

where $D^{*}$ are the 3D bounding boxes’ true dimensions, $\overline{D}$ are the mean dimensions of the objects of a specific class, n is the number of objects in the training batch, and $\delta$ is the predicted deviation value with respect to the average that the neural network predicts.

Figure 7 describes the architecture of the Dimensions Estimation Head. The number of fully connected units is (1280, 256, 3) and estimates the object dimensions.

3.3. Bird’s Eye View Bounding Box Center Estimation

The object’s visual characteristics and semantic category both influence its Bird’s Eye View map (e.g., a truck is longer than a car). Thus, estimating the corresponding Bird’s Eye View bounding box center of a detected object can be issued from a deep learning perspective [125].

As shown in Figure 8, the general architecture of the Bird’s Eye View Bounding Box Center Estimator is divided into encoder stage and decoder stage. The encoder stage is responsible for encoding the input features into the feature vector. The decoder stage is responsible for decoding the feature vector into the Bird’s Eye View centers ($t_x$,$t_z$).

As detailed in Figure 9, the Bird’s Eye View Center Estimator encoder stage is composed of five major branches, each branch acts as an encoder to its input feature. The visual feature encoder branch accepts image crops of vehicles detected in the frontal camera view as input, then extracts deep representations using the GhostNet network. This part of the model can extract semantic features from input images despite not knowing where the bounding box is in the scene. It generates a 1280 element feature vector. Each of the remaining four encoder branches generates a 256 element feature vector.

The first branch processes the estimated physical object dimensions, while the second branch takes the estimated object orientation as its input. The third branch accepts the 2D bounding box coordinates from the frontal camera view and the fourth branch takes the ratio between the estimated physical height of the object to the object’s visual height in the 2D bounding box. This ratio plays the role of the initial guess of the object depth as shown in Figure 10.

Depending on the geometry of a pinhole camera, the distance between the object center and the camera (Z) is denoted in accordance with the details presented in Equation (5) [126]

$$Z=\frac{fH}{h}$$

(5)

where h is the height of the 2D bounding box, f is the camera focal length, and H is the actual physical height of the object.

As the exact actual physical height (H) is estimated to be ($d_y$), it can be used to give an initial guide to the depth, as explained in Equation (6) [127]

$$Z^{*}=\frac{f{d}_y}{h}$$

(6)

where $d_y$ is the estimated physical height and $Z^{*}$ is the guiding depth.

The four encoded feature vectors in addition to the GhostNet extracted features of Figure 8 are concatenated to generate one geometric-visual fused feature vector of size (256 × 4 + 1280 = 2304) elements and fed to the decoder network which estimates the center coordinate ($t_x$,$t_z$) of the Bird’s Eye View bounding box, as shown in Figure 11.

In this way, the above model depicted in Figure 8 accepts the full representative geometric-visual properties of the object and learns to predict the corresponding Bird’s Eye View bounding box center of the detected object. The full Bird’s Eye View center estimation model was trained on L2 Loss [124] to generate the translation loss ($L_{Translations}$), as the explanation provided in Equation (7):

$$L_{Translations}=\frac{1}{n}\sum {(T^{*}-T)}^2$$

(7)

where $T^{*}$ are the true Bird’s Eye View centers, n is the number of objects in the training batch, and T are the estimated Bird’s Eye View centers.

3.4. GhostNet as an Off-the-Shelf Lightweight Feature Extractor

Feature maps are spatial maps created by utilizing a regular convolution layer to inputs. Based on learned convolution filter weights for that layer, these feature maps are responsible for preserving certain feature-rich characterizations of the input image. By examining extracted features in standard convolution layers for evidence that they possess a particular pattern, it is observed that there are multiple close copies of special intrinsic extracted features in the entire set of feature maps developed by the convolution operation. The generation of which becomes redundant since the convolution operation is quite computationally expensive. Accordingly, the term “Ghost Feature Maps” [121] refers to these redundant duplicates. The actual reason for working on GhostNet is limiting the number of parameters, FLOPs (FLoating point OPerations), and becoming very close to the benchmark accuracy obtained by utilizing the original feature maps. GhostNet generates a percentage of the overall output feature maps, whereas the rest are generated by a simple linear operation. This low-cost linear operation leads to a significant reduction in parameters and FLOPs while maintaining nearly the same performance as the original baseline model [121]. The low-cost linear operation is crafted to be learnable and input-dependent, allowing it to be optimized using backpropagation in the backward pass. The effectiveness of this network as a feature extractor [128,129,130] is proved to ideally fit the Orientation-Dimensions Estimator, and the Bird’s Eye View Center Estimator, which results in overall only 14 million parameters for both models while competing with the state-of-the-art monocular 3D object detection models.

4. Experimental Work

4.1. Implementation Details

As mentioned in the previous section, the proposed MonoGhost network estimates the object’s 3D properties from 2D bounding box extracted from monocular images depending on two main stages: The first stage, shown in Figure 5 above, is composed of a feature extractor and three fully connected branches that share the extracted features’ pool. This first stage is responsible for estimating the object’s 3D dimensions as well as its orientation. The second stage, shown in Figure 8 above, is composed of a feature extractor and four 1D convolution branches, which work as encoders for the supplied visual-geometric features. All the encoded features are fused and fed to a 1D convolution network to work as a decoder that finally predicts the Bird’s Eye View center of the object.

4.2. Dataset

The KITTI [22] dataset is a widely available open-source dataset for accurately assessing learning-based methodologies on driving scenes. It is believed to depict accurate representations of autonomous driving scenarios. There are 7481 training images and 7518 testing images in total. Depending on the occlusion, truncation level and the appeared height of the 2D bounding box of object instances, the test metric is divided into easy, moderate, and hard cases [22]. The 3D Object Detection and Bird’s Eye View benchmarks are available for testing the proposed method’s 3D detection task.

4.2.1. Data Augmentation

Data augmentation techniques were utilized to increase the training dataset, improve the training process, and avoid overfitting. The adopted techniques are random Gauss noise, optical distortion with probability 0.5, and random fog with probability 0.8.

4.2.2. Preprocessing

Applying any complex preprocessing method on the dataset was avoided. Instead, the only preprocessing step was applying resize with padding on the cropped 2D image of the detected object. The aim of this step is to preserve the object properties before being propagated to the Orientation-Dimensions Estimator or to the Bird’s Eye View Center Estimator, rather than losing object properties by using scaling only as illustrated by Figure 12. The Orientation-Dimensions Estimator and the Bird’s Eye View Center Estimator expect to receive images with fixed dimensions 112 × 112 × 3.

4.3. Training

The training process was performed for the MonoGhost network on a Geforce RTX 3060 Ti 8G. The two branches of the network were trained independently. The optimizer was chosen to be AdamW [131] with weight decay 1 $\times {10}^{-3}$ for both branches.

4.3.1. Orientation-Dimensions Estimator

The Orientation-Dimensions Estimator was trained on batch size of 200 for 250 epochs. The learning rate was set initially at 1 × 10${}^{-4}$. The scheduler was set to be reduced on plateau by a factor of 0.1, with patience 10, and threshold 1 × 10${}^{-4}$. The optimizer was chosen to be AdamW with weight decay 1 × 10${}^{-3}$.

4.3.2. Bird’s Eye View Center Estimator

The Bird’s Eye View Center Estimator was trained on batch size of 200 for 500 epochs. The learning rate was set initially at 1 × 10${}^{-3}$. The scheduler was set to cosine annealing with a maximum number of iterations at 4, and a minimum learning rate of 5 × 10${}^{-8}$.

5. Results and Discussion

As the proposed MonoGhost network is composed of two stages, and each one can be trained independently, the results of the proposed network can be validated on two different benchmarks, the first benchmark (https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d, accessed on 5 November 2023) is the object orientation, while the second benchmark (https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=bev, accessed on 5 November 2023) can be used to verify the results of the complete Bird’s Eye View estimation. For both benchmarks, an off-the-shelf 2D object detector Faster RCNN [119] is utilized to supply the 2D bounding boxes’ coordinates.

The first contribution of MonoGhost network is proposing a deep learning architecture (Bird’s Eye View Center Estimator) for estimating Bird’s Eye View bounding box of the detected object, depending on encoding the object geometric and visual features using 1D convolution then fusing the encoded features to decode the object Bird’s Eye View translation center. As shown in Figure 13, the average error of Z estimation on the KITTI training dataset was 0.682 m, with standard deviation 0.732 m, and the maximum error was 11.9 m; while the average error of X estimation was 0.243 m, with standard deviation 0.3035 m, and the maximum error was 6.658 m, as stated in Figure 14.

Table 1. MonoGhost network orientation score on KITTI benchmark. An increase in the measured accuracies, as indicated by the ⇑ up arrows, corresponds to better results.

Benchmark	Easy ⇑	Moderate ⇑	Hard ⇑
Car (Detection)	90.79%	83.33%	71.13%
Car (Orientation)	90.23%	82.27%	69.81%

shows the proposed MonoGhost network results for orientation on KITTI benchmark.

The values presented in Table 1 characterize the accuracy of orientation estimation by the MonoGhost network. It was assessed using the Average Orientation Similarity (AOS) metric, as defined in [22], which incorporates both the average precision (AP) of the 2D detector and the average cosine distance similarity for azimuth orientation.

A sample of the Bird’s Eye View results of testing MonoGhost network on KITTI samples is shown in Figure 15.

The second contribution of MonoGhost network is preserving the performance of the stable object’s depth estimation for KITTI hard object cases without sacrificing the orientation prediction accuracy. Figure 16 shows the average depth estimation error plotted in 10 m intervals. It is shown that the implemented MonoGhost network retains a rather stable error across all the intervals, and scores the best error for ground truth distance further than 60 m. This insight is assured by scoring 15.01% on KITTI hard object cases, which surpasses the performance of the most-known state-of-the-art monocular detection networks. Moreover, Table 1 shows a stable orientation estimation on KITTI’s moderate and hard object cases.

The last contributions of MonoGhost network are the simple design of MonoGhost network, which is composed of low computational burden operations and the selection of an efficient lightweight feature extractor. Thus, it is crucial to measure the time performance in order to maintain the real-time requirements. By evaluating the inference time of the model on Geforce RTX 3060 Ti 8G, it shows 0.033 s on average per batch composed of 70 objects; on GeForce GTX 1050 Ti it achieves 0.058 for a batch of 70 objects. The obtained inference time benchmarks ensure real-time operation even while under-utilizing relatively cheap, not powerful GPUs.

Table 2. A comparison between the Bird’s Eye View benchmark scores between the most known state-of-the-art monocular 3D object detection networks and the proposed MonoGhost network as submitted on KITTI car Bird’s Eye View benchmark leaderboard (https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=bev, accessed on 5 November 2023). Notably, the rank column represents a global rank, encompassing all detection techniques, including LiDAR-based, stereo-based, depth-aided, and single-image-based methods. The direction of the arrows in each column signifies the trend of increasing model performance.

Method	Rank ⇓	Easy ⇑	Moderate ⇑	Hard ⇑
PGD-FCOS3D [23]	406	26.89%	16.51%	13.49%
KM3D-Net [24]	409	23.44%	16.20%	14.47%
D4LCN [101]	410	22.51%	16.02%	12.55%
MonoPair [112]	413	19.28%	14.83%	12.89%
Decoupled-3D [109]	414	23.16%	14.82%	11.25%
QD-3DT [110]	415	20.16%	14.71%	12.76%
SMOKE [25]	416	20.83%	14.49%	12.75%
RTM3D [108]	417	19.17%	14.20%	11.99%
Mono3D_PLiDAR [99]	418	21.27%	13.92%	11.25%
M3D-RPN [114]	420	21.02%	13.67%	10.23%
MonoCInIS [111]	424	22.28%	11.64%	9.95%
MONO3D [132]	-	5.22%	5.19%	4.13%
Proposed MonoGhost network	405	24.91%	16.73%	15.01%

presents a comprehensive comparison of the Bird’s Eye View results scored between the proposed MonoGhost network and well-known state-of-the-art monocular 3D object detection networks, including PGD-FCOS3D [23], ImVoxelNet [133], SMOKE [25], and M3D-RPN [114], as listed on the KITTI leaderboard. The KITTI Bird’s Eye View benchmark employs the PASCAL criteria [134] to evaluate detection performance [22]. The KITTI Bird’s Eye View Benchmark defines difficulties based on the following criteria:

• Easy: Minimum bounding box height: 40 pixels, Maximum occlusion level: Fully visible, Maximum truncation: 15%.
• Moderate: Minimum bounding box height: 25 pixels, Maximum occlusion level: Partly occluded, Maximum truncation: 30%.
• Hard: Minimum bounding box height: 25 pixels, Maximum occlusion level: Difficult to see, Maximum truncation: 50%.

These difficulty levels provide a clear framework for assessing the performance of the proposed MonoGhost network in a range of challenging scenarios.

Table 3. A comparison between the most-known state-of-the-art monocular 3D object detection networks and the proposed MonoGhost network depending on the utilized GPU, inference time, and the adopted method. The direction of the arrows in each column signifies the trend of increasing model performance.

Method	Time ⇓	Hardware	Adopted Aproach	CUDA Cores ⇓	GPU Memory ⇓
PGD-FCOS3D [23]	0.03 s	4xGTX 1080Ti	Single-Image-Based Approach	4 × 3584	4 × 11 GB
KM3D-Net [24]	0.04 s	1xGTX1080Ti	Single-Image-Based Approach	1 × 3584	1 × 11 GB
D4LCN [101]	0.2 s	4xTesla v100 GPUs	Depth-Aided Approach	1 × 5120	1 × 32 GB
MonoPair [112]	0.057 s	1xGTX1080Ti	Single-Image-Based Approach	1 × 3584	1 × 11 GB
Decoupled-3D [109]	0.08 s	Not mentioned	Single-Image-Based Approach	-	-
QD-3DT [110]	0.03 s	RTX 2080Ti	Single-Image-Based Approach	1 × 4352	1 × 11 GB
SMOKE [25]	0.03 s	4xGeforce TITAN X GPUs	Single-Image-Based Approach	4 × 3072	4 × 12 GB
RTM3D [108]	0.05 s	2xGTX1080Ti	Single-Image-Based Approach	2 × 3584	2 × 11 GB
Mono3D_PLiDAR [99]	0.1 s	Not mentioned	Depth-Aided Approach	-	-
M3D-RPN [114]	0.16 s	1xGTX1080Ti	Single-Image-Based Approach	1 × 3584	1 × 11 GB
MonoCInIS [111]	0.13 s	1xGTX1080Ti	Single-Image-Based Approach	1 ×  3584	1 × 11 GB
Proposed MonoGhost network	0.03 s	1xRTX3060Ti	Single-Image-Based Approach	1 × 4864	1 × 8 GB

presents a detailed analysis of the inference time (time taken to process one frame) and the associated hardware deployment requirements for each method mentioned in Table 2. The table shows that the proposed MonoGhost method has an inference time of 0.03 s on RTX3060TI GPU. This time is shared with three other techniques (PGD-FCOS3D [23], QD-3DT [110], and SMOKE [25]). It is worthwhile to note here that these three methods employed much higher and more expensive GPUs with larger amounts of GPU memories and CUDA cores to achieve the same inference time as the proposed MonoGhost method. It is noteworthy that QD-3DT [110], despite utilizing an RTX2080TI GPU with a relatively lower number of CUDA cores, compensates for this limitation with a significantly higher GPU memory capacity. As a result, it boasts a markedly more powerful hardware configuration compared to the proposed MonoGhost network, which utilizes an RTX3060Ti GPU. This heightened GPU capacity empowers QD-3DT [110] to achieve comparable inference times to the proposed MonoGhost network. Notably, even when QD-3DT [110] employs the more powerful RTX2080TI, its performance ranking remains significantly lower than that of MonoGhost on KITTI Bird’s Eye View Leaderboard. This analysis underscores the superior performance of the proposed MonoGhost network, both in terms of inference time and the necessary hardware configuration to achieve efficient inference. The results highlight the MonoGhost network as a compelling choice for real-time applications that can make use of hardware environments with limited resources. It not only excels in terms of inference speed but also minimizes hardware demands, further enhancing its suitability for resource-constrained settings.

By combining the Bird’s Eye View results obtained in Table 2 with the inference time comparison from Table 3, the MonoGhost network demonstrates its potential to achieve a stable detection score while maintaining a very reasonable inference time.

Notably, the above results from Monocular 3D object detection methods, in Table 2 and Table 3, employ cheap camera sensors and generate depth information by computation or estimation. Their ranks in the KITTI Bird’s Eye View benchmark (https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=bev, accessed on 5 November 2023) range from 405 to 424 and their accuracies range from 4% to 26%. This can be achieved without the need for powerful/expensive hardware sensor setup such as the case with LiDAR-based 3D detection methods (e.g., IA-SSD [135], RangeIoUDet [136], StructuralIF [137], and RangeDet [138]). These LiDAR-based 3D detection methods employ comparatively very expensive hardware, require large computational powers, and inherently generate depth perception for the surrounding environment. These methods rank from 68 to 195 in the KITTI Bird’s Eye View benchmark (https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=bev, accessed on 5 November 2023) and their accuracies range from 82% to 93%. These two accuracy ranges are completely distinct and can not be compared together without considering the other underlying hardware/computational factors.

In order to perform a quick ablation study of the Bird’s Eye View Bounding Box Center Estimation stage of the proposed MonoGhost network, each of the five inputs to the Bird’s Eye View Bounding Box Center Estimation stage were zeroed in turn (one at a time) in order to establish their effect on/contribution to the final output of the system. The results indicated that removing any one of these inputs yielded indefinite NaN (=Not a number) results. This implies that the removal of any of these inputs requires a substantial change in the architecture of the proposed network.

The above results show that the proposed MonoGhost is a highly suitable solution for detection of class “Car” in autonomous driving systems. It is worthwhile to note here that classes other than “Car” have been considered in only five of the techniques provided in Table 2 (PGD-FCOS3D [23], D4LCN [101], MonoPair [112], QD-3DT [110], and M3D-RPN [114]). The proposed MonoGhost network can be employed to detect classes other than cars (e.g., pedestrians, cyclists, etc.); however, this is outside the scope of this paper. Please Note that the KITTI ranking (https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=bev, accessed on 5 November 2023) [22] was conducted based on the moderate score of proposed model. The MonoGhost network proves that the orientation estimation depending on MultiBin discrete–continuous methodology extensively outperforms other methods by utilizing a lightweight feature extractor.

6. Conclusions

This paper presented a novel lightweight architecture to estimate the Bird’s Eye View bounding boxes for known object classes from a single image. The method decouples the 3D bounding box estimation problem into two separate tasks, by first estimating the object’s geometric features presented by the orientation and dimensions then fusing these estimates with the object visual features in order to output a stable 3D bounding box. The proposed Monocular 3D detection MonoGhost network achieves promising results in terms of accuracy by achieving 24.91% on easy cases, 16.73% on moderate cases, and 15.01% on hard cases of KITTI Bird’s Eye View benchmark (https://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=bev, accessed on 5 November 2023), with an average inference time of 0.033 s per batch of 70 objects. The proposed MonoGhost network has the potential of integrability with cutting-edge 2D object detection platforms in order to be deployed in autonomous vehicles and in robotic navigation.