Identification and Positioning Method of Bulk Cargo Terminal Unloading Hopper Based on Monocular Vision Three-Dimensional Measurement

Abstract

Rapid identification and localization of dry bulk cargo hoppers are currently core issues in the automation control of gantry cranes at dry bulk terminals. The current conventional method relies on LiDAR systems for the identification and positioning of bulk unloading hoppers. However, this approach is complex and costly. In contrast, GPS-based positioning solutions for bulk unloading hoppers are prone to damage due to the vibrations generated during the operation process. Therefore, in this paper, a hopper localization system based on monocular camera vision is proposed to locate the position of the bulk unloading hopper. The hopper identification and localization process are divided into three stages. The first stage uses the improved YOLOv5 model to quickly and roughly locate the hopper target. The second stage uses morphological geometrical features to locate the corner points of the hopper target. The third stage determines the three-dimensional coordinates of the hopper target by solving the position of the corner points in the world coordinate system through the PnP (Perspective-n-Point) algorithm. The experimental results show that the average positioning accuracy of the coordinates of the method is above 93%, demonstrating the accuracy and effectiveness of the method.

1. Introduction

The loading and unloading speed of cargo is a key metric in dry bulk terminals [1], and the schematic diagram of the gantry crane ship unloading operation is shown in Figure 1 [2]. At present, the efficiency of the gantry crane loading and unloading operation mainly depends on the experience of the operator [3]. Therefore, it is necessary to position the bulk unloading hopper to assist the grab in unloading the ship. To measure the relative position between the grab and the bulk unloading hopper, there are two main engineering application schemes:

1. Using a contact measurement approach, sensors such as attitude sensors and GPS devices are deployed on the upper part of the bulk hopper for direct positional measurement. However, these sensors are highly susceptible to vibration, impact, and shock, leading to potential failures. Moreover, the bulk unloading hopper is mobile and lacks inherent power and communication networks, making the deployment of such sensors complex.

2. Using a non-contact measurement approach typically involves deploying LiDAR capable of three-dimensional measurement for the bulk unloading hopper. However, LiDAR is expensive and easily affected by material dust and rain, which severely compromise measurement accuracy. Additionally, LiDAR’s limited field of view results in a small effective measurement range, thereby impacting overall measurement efficiency [4].

Based on this, this paper proposes a monocular camera vision localization system for locating the bulk unloading hopper to improve the efficiency of gantry crane unloading operations. In this paper, we use the improved YOLOv5 model with image processing for bulk unloading hopper corner point localization and apply the PnP algorithm to measure the target’s world coordinates. The contributions of this paper are mainly reflected in the following aspects:

1. We propose a device and approach based on monocular vision for three-dimensional measurement, achieving bulk unloading hopper recognition and spatial positioning solely through a single camera.

2. We have improved the traditional YOLOv5 model and proposed a balanced accuracy-speed YOLOv5 model for the coarse localization of bulk unloading hoppers. This model is combined with morphological image processing methods to accurately determine the pixel coordinates of the hopper corners.

3. We propose a method for solving the three-dimensional coordinates of the bulk unloading hopper using a monocular camera based on the PnP algorithm. By utilizing the pixel coordinates of the four corners of the bulk unloading hopper, we directly determine its spatial coordinates.

The detailed composition of this paper is as follows: In Section 2, we discuss related work, including target detection algorithms and key point detection techniques based on machine vision. In Section 3, we introduce the proposed method for bulk unloading hopper localization in detail. Section 4 presents experimental results based on real data. Finally, Section 5 summarizes this paper.

2. Related Works

Vision-based measurement (VBM) systems are capable of obtaining wealth of information through analyzing images [5], and with the development of relevant visual inspection algorithms and the improvement of hardware arithmetic power, solutions using vision-based measurement are becoming more and more popular in ports. Ngo et al. [6], relied on a vision system to propose a solution for lane line detection in container terminals. Ji et al. [7], used vision systems combined with deep learning models to detect truck bodies and prevent truck lifting accidents. Lee [8] installed at least four cameras on cranes to assist in the detection and identification of the corner cast of the container. However, the identification of objects using only target detection does not tell the positional information of the target. Therefore, it is necessary to solve the 3D target coordinate information through the 2D image of the 3D target, and there are two common methods in deep learning: (1) direct regression methods; (2) key point solving coordinates.

Direct Regression Approach: Xu et al. [9], introduced a generalized framework to extend the 2D object detector to 3D object detection, employing End-to-end learning (ETL) for estimating the location of 3D objects. This approach uses an external network as at least one component of its framework, which allows for an inherent disconnect in component learning and system complexity. On this basis, Brazil et al. [10], proposed a Monocular 3D Region Proposal Network (M3D-RPN) to generate both 2D and 3D object candidate frames and design a novel depth-aware convolutional layer (CNN) to learn spatial features. Haq et al. [11], utilized geometric constraints in 3D and 2D detection spaces to predict 3D objects gained performance improvement. And these algorithms are usually too complex and have relatively poor real-time performance.

Key point coordinates localization: using known world coordinates, the position information of the target is obtained by solving the mapping relationship between the world coordinates of the target and the pixel coordinates. Mi et al. [12] used a triangulation method for the image after Zhang’s calibration [13] according to the characteristics of the pinhole camera, and utilized machine vision to realize the real-time orientation of the container’s three-dimensional pose through the center offset displacement of the locking hole. The camera position in the above literature is relatively fixed and does not satisfy the scenario of camera movement in this paper. The Perspective-n-Point (PnP) algorithm is commonly used to solve the transformation of pixel coordinates to world coordinates. Mi et al. [14], used two-stage PnP to solve the camera position and marker position; Shan et al. [15], solved the pose information of the object by the PnP algorithm; Li et al. [16], used PnP to solve the marker’s pose in the camera coordinate system. All of the above papers use target markers to improve the detection accuracy of the target, while the target in this paper has a simple structure, which can directly locate the coordinates of the target’s corner points, and the coordinate information can be known without fixing the target marker. For example, Rad et al. [17], predicted the 2D projections of the corner points of the target’s 3D boundaries from segmented 2D images, which is robust to partially occluded targets. Tekin et al. [18], proposed a similar scheme using the yolo architecture, with an innovative confidence scoring system that dispenses with the need for subsequent refinement steps. Ma et al. [19], used neural networks and greedy algorithms to directly regress the angular point information of the target. However, the accuracy of pose estimation is largely dependent on the quality of key point detection, and the quality of extracted features plays a key role in the performance of key point detection. Therefore, improving the quality of extracted features is still crucial for accurate key point detection as well as reliable attitude estimation results.

In addition to detection accuracy, the speed of target detection is also an important performance metric at bulk cargo terminal operations. YOLOv5 [20], due to its reliable accuracy and fast single-stage detection speed, has become a mainstream choice. However, the original YOLOv5 model tends to exhibit low frame rates when deployed on edge computing devices and when performing synchronous inference across multiple video streams, posing significant safety risks for industrial production and monitoring. In response, Zhang et al. [21], inspired by ShuffleNetv2 [22], have enhanced YOLOv5 by removing the focus layer and replacing the original feature extraction network with the ShuffleNetv2 algorithm, and pruning the neck part of the network. This effectively reduces the size of the original YOLOv5 model while significantly increasing detection speed. Li et al. [23], by integrating YOLOv5 with two lightweight models, ShuffleNet and MobileNet, significantly reduced the size of the model. However, this modification led to a notable decline in detection accuracy. Thus, effectively modifying YOLOv5 to balance detection accuracy with speed, to meet the computational demands of multiple automated handling devices at bulk cargo terminals on resource-limited devices, has become a critical challenge.

3. Our Method

In the process of automated loading and unloading operations at bulk cargo terminals, it is necessary to accurately position the bulk unloading hoppers. Fixed cameras, which have a wide field of view to cover the entire operation scene, often encounter challenges. The weak targets within the wide and complex scenes at the bulk terminal contain limited feature information, leading to significant deviations in target detection and identification based on visual neural networks.

Additionally, the GPS-based bulk unloading hopper positioning method may result in structural damage to the GPS equipment due to the violent vibrations caused by the collision of the grab bucket. Therefore, to improve the accuracy and robustness of bulk unloading hopper positioning during automated loading and unloading operations at bulk cargo terminals, this paper proposes a three-stage bulk unloading hopper offset positioning method.

3.1. System Design

The bulk cargo terminal loading and unloading operation process involves the use of a variety of equipment, mainly including the gantry crane and bulk unloading hopper, as shown in Figure 2. GPS is installed on the gantry for obtaining real-time world coordinate positioning of the key points of the gantry during operation. There are two GPS devices on the gantry crane elephant trunk: GPS1 $(\mathrm{X},\mathrm{Y},\mathrm{Z})$ and GPS2 $({\mathrm{X}}^{\prime },{\mathrm{Y}}^{\prime },{\mathrm{Z}}^{\prime })$ At the same time, a self-vertical camera is installed at the end of the maintenance platform of the crane, directly above the spreader, to obtain a visual image of the entire process of automated loading and unloading. By using GPS1 and GPS2, the camera’s coordinates in the world coordinate system can be calculated as $({\mathrm{X}}_{\mathrm{c}},{\mathrm{Y}}_{\mathrm{c}},{\mathrm{Z}}_{\mathrm{c}})$, as well as the direction vector $\overrightarrow{\mathit{v}}$ of the camera’s x-axis in the world coordinate system.

Due to the inherent shortcomings of visual cameras, accurately locating small targets in complex scenes, such as loading and unloading operations at bulk cargo terminals, is challenging. The 3D coordinate positioning method based solely on monocular machine vision can be significantly affected by pixel-level deviations in target recognition results, which seriously impacts the accuracy of automated loading and unloading operations at bulk cargo terminals.

In the automatic loading and unloading process of bulk cargo terminals, ${\mathrm{O}}_1$, ${\mathrm{O}}_2$, ${\mathrm{O}}_3$, and ${\mathrm{O}}_4$ are the four corner points of the bulk unloading hopper, ${\mathrm{O}}_{\mathrm{H}}$ is the center point of the bulk unloading hopper, and its world coordinates are $({\mathrm{X}}_{\mathrm{H}},{\mathrm{Y}}_{\mathrm{H}},{\mathrm{Z}}_{\mathrm{H}})$, where ${\mathrm{O}}_{\mathrm{P}}$ is the predicted center point of the bulk unloading hopper, and its world coordinates are $({\mathrm{X}}_{\mathrm{P}},{\mathrm{Y}}_{\mathrm{P}},{\mathrm{Z}}_{\mathrm{P}})$, as shown in Figure 3.

The bulk unloading hopper prediction accuracy is calculated considering the geometric error, combined with the bulk unloading hopper size for the definition of the percentage of accuracy. For the definition of error distance as shown in Equation (1), the distance error between the predicted bulk unloading hopper center point and the real bulk unloading hopper center point is calculated, which is calculated by the Euclidean distance formula.

$$\begin{array}{c}d=\sqrt{{\left({\mathrm{X}}_{\mathrm{P}}-{\mathrm{X}}_{\mathrm{H}}\right)}^2+{\left({\mathrm{Y}}_{\mathrm{P}}-{\mathrm{Y}}_{\mathrm{H}}\right)}^2+{\left({\mathrm{Z}}_{\mathrm{P}}-{\mathrm{Z}}_{\mathrm{H}}\right)}^2}\end{array}$$

(1)

In order to determine the reference dimension of the bulk unloading hopper, the side length L of the hopper is a reasonable choice. If the hopper is square, half of the diagonal is taken as the reference scale, which is shown as ${\mathrm{L}}_{\mathrm{r}\mathrm{e}\mathrm{f}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}}$, represents the maximum extent of the hopper.

$${\mathrm{L}}_{\mathrm{r}\mathrm{e}\mathrm{f}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}}={\displaystyle \frac{\sqrt{2}\times L}{2}}$$

(2)

Predictive positioning accuracy $\mathrm{P}$ is designed to be inversely proportional relationship, i.e., the larger the error, the lower the precision.

$$\mathrm{P}=\left\{\begin{array}{cc}(1-{\displaystyle \frac{d}{{\mathrm{L}}_{\mathrm{r}\mathrm{e}\mathrm{f}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}}}})\times 100\%& \mathrm{ },\mathrm{d}>{\mathrm{L}}_{\mathrm{r}\mathrm{e}\mathrm{f}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}}\\ 0& ,{\mathrm{d}\le \mathrm{L}}_{\mathrm{r}\mathrm{e}\mathrm{f}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}}\end{array}\right.$$

(3)

Taking the bulk cargo terminal of Tianjin Port as an example, the side length L of the bulk unloading hopper is 7 m, and the predicted positioning accuracy under this parameter $\mathrm{P}$ is expressed in Equation (4).

$$\mathrm{P}=\left(1-{\displaystyle \frac{\sqrt{2}\mathrm{d}}{7}}\right)\times 100\mathrm{\%}$$

(4)

At the Tianjin Port operation site, the maximum allowable operational error between the center points of the grab and the bulk unloading hopper is 50 cm, while the actual maximum average error for manual operations can reach 35 cm. According to the calculation based on Formula (4), the allowable operational accuracy for the center points of the grab and the hopper at Tianjin Port is 90%, while the actual average operational accuracy for manual operations reaches as high as 93%. Therefore, to ensure accuracy and safety, this paper adopts the average operational accuracy of manual operations, which is 93%, as the requirement for predictive positioning accuracy of automated bulk cargo terminal handling equipment.

At the same time, the improved neural network recognition algorithm for small and weak targets will significantly increase the processing time, which is intolerable in the automated loading and unloading process of bulk cargo terminals and will cause great safety hazards in the operation process.

Therefore, we have designed a three-stage bulk unloading hopper positioning method based on machine vision. The main method architecture is shown in Figure 4. In the coarse positioning stage, the bulk unloading hopper is identified by a lightweight and efficient visual neural network during operation, and the cropped target is passed to the fine positioning stage. In the fine positioning stage, a linear fitting method is used to determine the exact position of the target corner within the coarse positioning detection frame according to the geometric characteristics of the target. This position is then mapped to the coordinate system of the original image, providing accurate input data for the subsequent three-dimensional coordinate solving stage. In the three-dimensional coordinate solving stage, the image coordinates are back-projected to the world coordinate system using the camera’s intrinsic and extrinsic parameter matrices, and the deviation between the grab and the bulk unloading hopper is calculated.

3.2. The Coarse Positioning Stage

For the detection task during automated loading and unloading operations at bulk cargo terminals, achieving real-time detection while maintaining accuracy is crucial. This necessitates lightweight optimization of the model to strike a balance between ensuring accuracy and preventing significant degradation. Such optimization is essential to facilitate real-time synchronous processing of video streams from multiple gantry cranes in future implementations.

YOLOv5, proposed by Jocher, G. et al. [20] adopts the concept of one-stage object detection, we applied YOLOv5 for detecting the position of the bulk unloading hopper itself in the actual image and its network structure is shown in Figure 5. The YOLOv5 network architecture consists of three main components: the backbone, neck, and head. The CBS module comprises a convolutional layer followed by batch normalization and SiLU activation. The CSP1_X module evolves from CSPNet and includes CBL modules, several residual modules, convolutional layers, and concatenation operations, with ‘X’ indicating the number of such modules. Similarly, the CSP2_X module is also derived from CSPNet, consisting of a convolutional layer and ‘X’ number of CBL modules. It can balance the detection accuracy and detection speed.

YOLOv5 includes five different models: YOLOv5n, YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x. The primary differences among these models lie in the depth and width of the network, as well as variations in the number of channels. The designations n, s, m, l, and x represent models of different sizes, with ‘n’ being the smallest and ‘x’ the largest. And the official performance comparison of each version on the COCO public dataset is shown in

Table 1. Performance of YOLOv5 versions on COCO dataset [20].

Model	[email protected]/%	Speed/ms	Params/M
YOLOv5n	45.7	45	1.9
YOLOv5s	56.8	98	7.2
YOLOv5m	64.1	224	21.2
YOLOv5l	67.3	430	46.5
YOLOv5x	68.9	766	86.7

. [20] Although YOLOv5x achieves the best detection performance, its high computational complexity leads to a slower inference time due to its deep network architecture. In contrast, the lightweight model is more suitable for industrial applications ported to embedded devices. Therefore, we choose YOLOv5n as the base model in the coarse localization phase.

3.2.1. The Shufflenet V2 Optimized Yolov5 Backbone Network

We adopt a strategy of alternately stacking the base and downsampling modules of ShuffleNet V2 [22] to replace the original C3 and Conv modules in YOLOv5 to reduce model size and increase model speed. In the architecture of ShuffleNet V2, researchers invoke an innovative channel splitting technique that splits the input feature channel (c) into two branches, each of which possesses (c-c′) and c′ channels, respectively, where c′ usually takes the value of half of c. This splitting operation is to avoid the use of group convolution, thus reducing the Memory Access Cost (MAC) [22]. The basic module and down sampling module network structure is shown in Figure 6 and Figure 7.

Among these two branches, one remains unchanged as a residual connection (Identity branch), while the other is processed by three convolutional layers. These layers include two 1 × 1 convolutions and a 3 × 3 Depthwise Convolution (DWConv), all of which maintain the same number of input and output channels to ensure consistent channel widths. It is worth noting that these 1 × 1 convolutions do not use group convolution to minimize the increase in MACs caused by group convolution, and because the channel splitting operation itself already creates two separate channel groups.

The processed branches are subsequently merged through a concatenation (Concat) operation, which increases the number of channels in the network. This step ensures that information between different branches can be exchanged through the subsequent Channel Shuffle operation, which physically reorders the channels. The Channel Shuffle operation not only facilitates the flow of information between different groups but also enhances the network’s expressiveness without adding extra computational burden. Ultimately, through this channel splitting and shuffling, ShuffleNet V2 efficiently integrates and exchanges information while maintaining high computational efficiency, achieving an excellent balance of speed and accuracy across various levels of computational complexity.

To improve detection accuracy while ensuring the model remains lightweight, we adopt a strategy of alternately stacking the base and downsampling modules of ShuffleNet V2 to replace the original C3 and Conv modules in YOLOv5, thereby forming an efficient network structure. This design not only reduces the computational burden of the model but also enhances its performance in target detection through precise feature extraction. In this revised structure, the base module of ShuffleNet V2 is responsible for extracting rich feature information, while the downsampling module reduces the spatial resolution of the feature map via a convolution operation with a stride of 2 and increases the number of channels to facilitate feature fusion and information transfer at different levels. This alternate stacking strategy enables the network to capture multi-scale target information while maintaining a low parameter count, resulting in improved performance in target detection tasks.

3.2.2. Gsconv Convolutional Module Optimized Yolov5 Neck Part

Despite upgrading the backbone network to ShuffleNet V2 to reduce the computational burden of the model while maintaining detection accuracy, the network still contains a large number of convolutional operations, which consume a significant amount of computational resources to some extent. Although ShuffleNet V2 reduces computational load through pointwise convolution and channel shuffling, it still exhibits notable deficiencies in the depthwise separable convolution process.

Firstly, the separation of spatial and channel convolutions in depthwise separable convolution results in the fragmentation of feature information during processing, adversely affecting feature representation and fusion. This issue is particularly pronounced in complex scenarios or tasks requiring high-precision feature extraction, leading to a significant decline in model performance.

Secondly, while ShuffleNet V2’s structural design simplifies the computational process, it does not fully address the problem of efficient feature fusion, still necessitating substantial computational resources to maintain performance.

Furthermore, in practical applications, especially when handling high-resolution images or requiring real-time processing, the inefficiency of ShuffleNet V2 remains prominent. Its computational cost and resource consumption cannot be overlooked when pursuing higher detection accuracy.

GSConv [24], proposed by Li, H. et al., combines the features of depthwise separable convolution and standard convolution. The core idea of GSConv is to connect the output feature maps of these two types of convolution operations. This connection aims to realize feature complementation and enhancement, thereby improving the overall effectiveness of the model in capturing and processing diverse information from the input data. Subsequently, by introducing the Shuffle mechanism, GSConv facilitates the contact of information between the deeply separable convolution and the standard convolution feature maps, thus significantly reducing the computational complexity of the model while maintaining its accuracy.

Deep separable convolution effectively reduces the number of parameters and the computational effort of the model by decomposing the standard convolution into two steps: deep convolution and point-by-point convolution. However, this decomposition also leads to the loss of information between feature channels because the feature maps of each channel are processed independently. In contrast, standard convolution is able to maintain the contact between channels, but the computational expense is high. GSConv reduces the computational burden of the model while maintaining the accuracy of the model by concatenating the output feature maps of the depth-separable convolution and the standard convolution, and realizing the rearrangement of the information between the channels by shuffling operations. This design not only retains the advantage of depth-separable convolution in computational efficiency, but also improves the expressive power of the model through the fusion of inter-channel information of standard convolution.

3.2.3. The Simplified Yolov5 Detection Head

In the original YOLOv5 model, the detection head consists of three different scale detection heads, each responsible for detecting targets at different scales. This design is to ensure that the model can effectively detect targets of large, medium and small scales. However, in the fixed camera operation scenario of a bulk terminal, the need for detection of small targets is relatively low because the scale of the bulk unloading hopper targets does not vary much. Therefore, in this paper, the detection head of YOLOv5 is simplified by removing the detection head for small target detection to reduce the complexity and computational burden of the model. The simplified YOLOv5 model will contain only two detection heads, which are responsible for the detection of medium-scale and large-scale targets, respectively.

3.2.4. The Improved Yolov5 Framework

By integrating the following elements: a backbone based on ShuffleNet V2 basic modules and down sampling modules, using GSConv convolutional modules to replace traditional convolutions in the neck part, and removing the small object detection head in the head part, we have constructed the YOLOv5-slim model.

Specifically, we reconstructed the backbone network by employing the basic modules and downsampling modules of ShuffleNet V2. This structural design aims to reduce the number of parameters and computational cost, making the network more compact and thus more effective on resource-constrained on-site devices. ShuffleNet V2 achieves significant reductions in computation and improvements in efficiency through the introduction of depthwise separable convolutions, making it particularly suitable for environments with limited computational power, such as mobile devices. However, the depthwise separable convolutions separate channel information during computation, adversely affecting the capability for feature extraction and fusion, thereby significantly reducing the accuracy of these processes.

While ShuffleNet V2 offers clear advantages in terms of parameter reduction and computational cost, its limitations in feature extraction and fusion remain a critical issue. Although depthwise separable convolutions effectively lower computational complexity, the separation of channel information during convolution reduces inter-channel information exchange, impairing the comprehensive representation of feature maps and consequently affecting the overall model performance. To address this shortcoming, we introduced the GSConv module in the neck network to replace traditional convolution modules.

The GSConv module enhances information exchange between different convolutional feature maps by rearranging channel orders. Specifically, GSConv promotes cross-channel information sharing and fusion by reordering the channels, thereby improving feature extraction and fusion. This modification not only enhances the comprehensive representation of feature maps but also significantly improves the overall performance of the model while maintaining a lower computational complexity. Through this approach, the GSConv module achieves a balance between model precision and inference efficiency, making it suitable for deployment on resource-constrained devices.

By adopting the basic modules and downsampling modules of ShuffleNet V2, we achieved a compact and efficient network structure. However, to address the limitations in feature extraction and fusion capabilities inherent in depthwise separable convolutions, we introduced the GSConv module. The GSConv module enhances inter-channel information exchange by rearranging channel orders, significantly improving feature extraction and fusion capabilities. Ultimately, this integration of ShuffleNet V2 and the GSConv module, both modified and incorporated, works synergistically, achieving an ideal balance between model precision and inference efficiency.

In the context of fixed camera operations at bulk cargo terminals, where the target scale variation is minimal, we further reduced the model complexity by removing the small object detection head. The structure of the proposed coarse localization network is shown in Figure 8.

Compared to the traditional YOLOv5 model, the YOLOv5-slim model has fewer parameters and lower complexity, while still balancing model accuracy and inference efficiency. This makes it suitable for the low-power recognition requirements of automated loading and unloading operations in the context of bulk cargo terminals.

With the proposed YOLOv5-slim lightweight neural network, we are able to recognize the bulk unloading hopper in the operation video in real time. The recognition slices of the bulk unloading hopper are then transmitted to the fine localization stage through post-processing to complete the corner recognition acquisition.

3.3. The Fine Positioning Stage

The location of the bulk unloading hopper detected using the deep learning model is shown in Figure 9. The edge features of the image are the key to extract the target corner point information, and due to the influence of lighting factors and the target’s own structure, there will be more noise in the edge features, and the error of directly determining the corner point coordinates is larger. Therefore, the Canny algorithm [25] extracts the edge information of the image and the binarized image after Gaussian filtering are considered comprehensively, and the coordinates of the corner are determined with the geometric features of the outline itself as the constraint conditions.

3.3.1. Extract Edge Information

Canny edge detection detects edge information in an image by computing the gradient magnitude and direction of each pixel. The Sobel operator, which has relatively low computational complexity [26], is used with Sobel kernels $G_x$ and $G_y$ to represent the gradients in the x and y directions, respectively. The convolution operation between the convolution kernel and the input image yields the gradients in the horizontal and vertical directions of the image. Here, $d_x$ represents the gradient in the horizontal direction, $d_y$ represents the gradient in the vertical direction.

$$\begin{gathered} {\mathrm{G}}_{\mathrm{x}}=\left[\begin{array}{ccc}-1& 0& 1\\ -2& 0& 2\\ -1& 0& 1\end{array}\right] {\mathrm{G}}_{\mathrm{y}}=\left[\begin{array}{ccc}-1& -2& -1\\ 0& 0& 0\\ 1& 2& 1\end{array}\right], \\ {d}_x=f\left(x,y\right)\times G_x(x, y){ d}_y=f\left(x,y\right)\times G_y(x, y) \end{gathered}$$

(5)

d denotes the gradient magnitude at point $\left(\mathrm{x},\mathrm{y}\right)$, and $\theta$ represents the gradient direction at point $\left(x,y\right)$.

$$\mathrm{d}=\sqrt{d_{\mathrm{x}}^2+d_{\mathrm{y}}^2} , \mathsf{\theta }=\arctan \left({\displaystyle \frac{d_{\mathrm{y}}}{d_{\mathrm{x}}}}\right)$$

(6)

By selecting high and low thresholds, weaker gradient edge pixels are filtered out, while preserving edge pixels with stronger gradient values. If an edge pixel’s gradient value is higher than the high threshold value, it is marked as a strong edge pixel. If an edge pixel’s gradient value is smaller than the high threshold value and larger than the low threshold value, it is marked as a weak edge pixel. If an edge pixel’s gradient value is smaller than the low threshold value, it will be suppressed. Sample the target dataset to determine the optimal threshold for Canny edge detection, ensuring the best edge extraction from 50 sampled images. In clear daylight conditions, the double threshold for Canny edge detection is set at (154, 208), as depicted in Figure 10.

The noise in the edge features extracted by the Canny algorithm primarily originates from the edge features within the target. To obtain cleaner edge features of the target, the binarized brightness values at corresponding coordinates are combined with the Canny edge image, following 2D Gaussian filtering and morphological expansion, using a bitwise sum operation, as shown in Figure 11 The formulas for 2D Gaussian filtering and binarization are provided in Equation (7).

$$\begin{gathered} {\mathrm{H}}_{\left(\mathrm{i}\mathrm{j}\right)}\mathrm{ }=\mathrm{ }{\displaystyle \frac{1}{2\mathsf{\pi }{\mathsf{\sigma }}^2}}\mathrm{e}\mathrm{x}\mathrm{p}\left(-{\displaystyle \frac{{\left({\mathrm{x}}_{\mathrm{i}}-{\mathrm{u}}_{\mathrm{x}}\right)}^2\mathrm{ }+\mathrm{ }{\left({\mathrm{y}}_{\mathrm{i}}-{\mathrm{u}}_{\mathrm{y}}\right)}^2}{2\mathrm{ }{\mathsf{\sigma }}^2}}\right), \\ \mathrm{d}\mathrm{s}\mathrm{t}\left(\mathrm{x},\mathrm{y}\right)=\left\{\begin{array}{cc}\mathrm{m}\mathrm{a}\mathrm{x}\mathrm{V}\mathrm{a}\mathrm{l}& ,\mathrm{i}\mathrm{f} \mathrm{s}\mathrm{r}\mathrm{c}\left(\mathrm{x},\mathrm{y}\right)>\mathrm{t}\mathrm{h}\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{h}\hfill \\ 0& ,\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array}\right. \end{gathered}$$

(7)

3.3.2. Hough Transform for Corner Detection

After image processing, the edge features of the source image are more completely preserved. However, since the image feature contours are not continuous, conventional corner point detection algorithms cannot be used. Instead, straight line fitting and the target’s geometric features are utilized to extract the corner coordinates.

• Employing the Hough transform for fitting to generate all longer segments, identifying the longest segment among them, and recognizing the line where this segment lies as an edge of the bulk unloading hopper. Figure 12a illustrates the segments fitted using the Hough transform, while Figure 12b depicts the position of the line containing the longest segment.
• Since the camera coordinates are not fixed, rotation may cause the bulk unloading hopper to appear as an irregular rectangle. Based on the slope of the longest line segment, find the longest line segment that is approximately perpendicular to it. The slope threshold range is set to [−5, 5].
• Based on the slopes of the two intersecting sides and the center of the image, the original image is divided into four parts. Using the slope of the known straight line as a constraint, find the other two symmetric edges. The slope range is set to [−0.05, 0.05].
• If the area of the composed closed rectangle is less than 60% of the predicted rectangular box, set the pixel values of the longest straight line fitted in step 1 to 0 and restart step 1. The result of straight line fitting is shown in Figure 12.

In the image after coarse localization and cropping, the pixel coordinate system defaults with the origin at the top-left corner of the image. For the target coordinates in the original image $\left({\mathrm{u}}_{\mathrm{i}},\mathrm{ }{\mathrm{v}}_{\mathrm{i}}\right)$, and the target coordinate in the roughly localized image frame $\left({\mathrm{u}}_{\mathrm{i}}^{\prime },{\mathrm{v}}_{\mathrm{i}}^{\prime }\right)$, the following relational equation is satisfied between the two:

$$\left\{\begin{array}{c}{\mathrm{u}}_{\mathrm{i}}={\mathrm{u}}_{\mathrm{i}}^{\prime }+{\mathrm{u}}_0\\ {\mathrm{v}}_{\mathrm{i}}={\mathrm{v}}_{\mathrm{i}}^{\prime }+{\mathrm{v}}_0\end{array}\right.$$

(8)

In addition, the $\left({\mathrm{u}}_0,\mathrm{ }{\mathrm{v}}_0\right)$ denotes the coordinates of the upper left corner point of the roughly localized image frame again in the original image. In a subsequent step, the transformed coordinates of each of the four corner points are used $\left({\mathrm{u}}_{\mathrm{i}},{\mathrm{v}}_{\mathrm{i}}\right)$ as inputs to the 3D coordinate solving stage to realize the mapping transformation from target pixel coordinates to world coordinates.

In practical work scenarios, algorithms may encounter situations where straight lines cannot be detected, leading to an inability to accurately determine the positions of the four corner points. When this situation occurs occasionally, the system automatically disregards the detection result. Instead, it utilizes the current coordinates from the down-ward-facing camera, yaw angle, and the coordinates from the last successful positioning to estimate the central position of the bulk material discharge hopper reasonably. This ensures the coherence of the overall system workflow. If this situation persists, the system will notify remote operators to intervene and ensure the normal operation of cargo positioning tasks.

3.4. The Three-Dimensional Coordinate Solving Stage

Given the known world coordinates of the camera $(X_C,Y_C,Z_C)$, the two-dimensional coordinates of the four corner points of the bulk unloading hopper in the camera image are $(u_1,v_1)$, $(u_2,v_2)$, $(u_3,v_3)$ and $(u_4,v_4)$. The height of the bulk unloading hopper in the field environment is known as $H$. The current orientation of the elephant trunk in the world coordinate system is calculated by GPS1 and GPS2, providing the direction vector of the x-axis in the world coordinate system as viewed in the camera image coordinate system, denoted as $\overrightarrow{v}$. This vector represents the direction of the x-axis in the camera view image coordinate system relative to the world coordinate system, assuming that the coordinates of the four corner points of the bulk unloading hopper in the world coordinate system are, respectively $(X_1,Y_1,H)$, $(X_2,Y_2,H)$, $(X_3,Y_3,H)$ and $(X_4,Y_4,H)$.

3.4.1. PNP Algorithm Inverse Solution for World Coordinates

Using the camera’s intrinsic matrix $K$ and the extrinsic matrix $\left[R|t\right]$, we back-project from image coordinates to world coordinates. First establish the connection between the camera coordinate system and the image coordinate system as shown in Equation (9).

$$\mathrm{s}\left(\begin{array}{c}{\mathrm{u}}_{\mathrm{i}}\\ {\mathrm{v}}_{\mathrm{i}}\\ 1\end{array}\right)=\mathrm{K}\left[\mathrm{R}|\mathrm{t}\right]\left(\begin{array}{c}{\mathrm{X}}_{\mathrm{i}}\\ {\mathrm{Y}}_{\mathrm{i}}\\ \mathrm{H}\\ 1\end{array}\right)$$

(9)

where $\mathrm{s}$ is the scale factor. Expanding Equation (10) can be obtained:

$$\mathrm{s}\left(\begin{array}{c}{\mathrm{u}}_{\mathrm{i}}\\ {\mathrm{v}}_{\mathrm{i}}\\ 1\end{array}\right)=\left(\begin{array}{ccc}{\mathrm{f}}_{\mathrm{x}}& 0& {\mathrm{c}}_{\mathrm{x}}\\ 0& {\mathrm{f}}_{\mathrm{y}}& {\mathrm{c}}_{\mathrm{y}}\\ 0& 0& 1\end{array}\right)\left(\begin{array}{cccc}{\mathrm{r}}_{11}& {\mathrm{r}}_{12}& {\mathrm{r}}_{13}& {\mathrm{t}}_{\mathrm{x}}\\ {\mathrm{r}}_{21}& {\mathrm{r}}_{22}& {\mathrm{r}}_{23}& {\mathrm{t}}_{\mathrm{y}}\\ {\mathrm{r}}_{31}& {\mathrm{r}}_{32}& {\mathrm{r}}_{33}& {\mathrm{t}}_{\mathrm{z}}\end{array}\right)\left(\begin{array}{c}{\mathrm{X}}_{\mathrm{i}}\\ {\mathrm{Y}}_{\mathrm{i}}\\ \mathrm{H}\\ 1\end{array}\right)$$

(10)

In the equation, ${\mathrm{f}}_{\mathrm{x}}$ and ${\mathrm{f}}_{\mathrm{y}}$ represent the focal lengths along the x-axis and y-axis of the image, respectively. ${\mathrm{c}}_{\mathrm{x}}$ and ${\mathrm{c}}_{\mathrm{y}}$ denote the coordinates of the principal point on the image plane, typically expressed in pixel coordinates. The camera extrinsic matrix $\left[\mathrm{R}|\mathrm{t}\right]$ combines the rotation matrix $\mathrm{R}$ and translation vector $\mathrm{t}$. These are used to describe the complete transformation from the world coordinate system to the camera coordinate system. In particular, the rotation matrix $\mathrm{R}$ represents the orientation transformation from the world coordinate system to the camera coordinate system. The direction vector of the x-axis in the world coordinate system is calculated from the camera view image coordinate system $\overrightarrow{\mathrm{v}}$ The calculation is performed by taking $({\mathrm{X}}_{\mathrm{v}},{\mathrm{Y}}_{\mathrm{v}},{\mathrm{Z}}_{\mathrm{v}})$ as the first column vector of the rotation matrix, but to form a legal rotation matrix, the following conditions need to be satisfied: the column vectors of the matrix must be unit vectors and the column vectors of the matrix must be orthogonal to each other.

Therefore, the first vector is first normalized to the unit vector ${\overrightarrow{\mathrm{d}}}_1$.

$${\overrightarrow{\mathrm{d}}}_1={\displaystyle \frac{\left({\mathrm{X}}_{\mathrm{v}},{\mathrm{Y}}_{\mathrm{v}},{\mathrm{Z}}_{\mathrm{v}}\right)}{\sqrt{{\mathrm{X}}_{\mathrm{v}}^2+{\mathrm{Y}}_{\mathrm{v}}^2+{\mathrm{Z}}_{\mathrm{v}}^2}}}$$

(11)

Choose an arbitrary vector that is not parallel to the first vector $\overrightarrow{\mathrm{u}}$ and compute the cross product of this vector with the first vector. The result of this cross product will be perpendicular to the original two vectors, ensuring orthogonality. The resulting vector is then normalized to obtain the second unit vector.

$${\overrightarrow{\mathrm{d}}}_2=\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{z}\mathrm{e}\left({\overrightarrow{\mathrm{d}}}_1\times \overrightarrow{\mathrm{u}}\right)$$

(12)

where normalize means dividing the vector by its modulus to ensure that it becomes a unit vector.

To ensure that the three vectors constructed are orthogonal to each other, compute ${\overrightarrow{\mathrm{d}}}_1$ the cross product of ${\overrightarrow{\mathrm{d}}}_2$ the cross product of the three vectors. This vector will naturally be orthogonal to the first two vectors. Normalize the resulting vector.

$${\overrightarrow{\mathrm{d}}}_3=\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{z}\mathrm{e}\left({\overrightarrow{\mathrm{d}}}_1\times {\overrightarrow{\mathrm{d}}}_2\right)$$

(13)

By following the above steps, three vectors that are orthogonal to each other and of unit length are obtained. These vectors form the three column vectors of the rotation matrix, each representing the direction of one axis in the new coordinate system. This rotation matrix ensures that lengths and angles remain invariant during coordinate transformations.

The rotation matrix $\mathrm{R}$ can be expressed as:

$$\mathrm{R}=\left(\begin{array}{ccc}{\overrightarrow{\mathrm{d}}}_1& {\overrightarrow{\mathrm{d}}}_2& {\overrightarrow{\mathrm{d}}}_3\end{array}\right)$$

(14)

Translation vector $\mathrm{t}$ represents the position of the camera coordinate origin in the world coordinate system. Knowing the position of the camera in the world coordinate system $({\mathrm{X}}_{\mathrm{C}},{\mathrm{Y}}_{\mathrm{C}},{\mathrm{Z}}_{\mathrm{C}})$ then $\mathrm{t}$ can be expressed as:

$$\mathrm{t}=\left(\begin{array}{c}{\mathrm{X}}_{\mathrm{C}}\\ {\mathrm{Y}}_{\mathrm{C}}\\ {\mathrm{Z}}_{\mathrm{C}}\end{array}\right)$$

(15)

Finalizing the extrinsic matrix $\left[\mathrm{R}|\mathrm{t}\right]$ is

$$\left[\mathrm{R}|\mathrm{t}\right]=\left(\begin{array}{cccc}{\overrightarrow{\mathrm{d}}}_1& {\overrightarrow{\mathrm{d}}}_2& {\overrightarrow{\mathrm{d}}}_3& \mathrm{t}\end{array}\right)=\left(\begin{array}{cccc}{\mathrm{r}}_{11}& {\mathrm{r}}_{12}& {\mathrm{r}}_{13}& {\mathrm{X}}_{\mathrm{c}}\\ {\mathrm{r}}_{21}& {\mathrm{r}}_{22}& {\mathrm{r}}_{23}& {\mathrm{Y}}_{\mathrm{c}}\\ {\mathrm{r}}_{31}& {\mathrm{r}}_{32}& {\mathrm{r}}_{33}& {\mathrm{Z}}_{\mathrm{c}}\end{array}\right)$$

(16)

Bringing the extrinsic matrix $\left[\mathrm{R}|\mathrm{t}\right]$ into the original equation, solving the equation for the four corner points of the bulk unloading hopper in the world coordinate system can be obtained:

$$\mathrm{s}\left(\begin{array}{c}{\mathrm{u}}_{\mathrm{i}}\\ {\mathrm{v}}_{\mathrm{i}}\\ 1\end{array}\right)=\left(\begin{array}{c}{\mathrm{f}}_{\mathrm{x}}\left({\mathrm{r}}_{11}{\mathrm{X}}_{\mathrm{i}}+{\mathrm{r}}_{12}{\mathrm{Y}}_{\mathrm{i}}+{\mathrm{r}}_{13}\mathrm{H}+{\mathrm{X}}_{\mathrm{c}}\right)+{\mathrm{c}}_{\mathrm{x}}\left({\mathrm{r}}_{31}{\mathrm{X}}_{\mathrm{i}}+{\mathrm{r}}_{32}{\mathrm{Y}}_{\mathrm{i}}+{\mathrm{r}}_{33}\mathrm{H}+{\mathrm{Z}}_{\mathrm{c}}\right)\\ {\mathrm{f}}_{\mathrm{y}}\left({\mathrm{r}}_{21}{\mathrm{X}}_{\mathrm{i}}+{\mathrm{r}}_{22}{\mathrm{Y}}_{\mathrm{i}}+{\mathrm{r}}_{23}\mathrm{H}+{\mathrm{Y}}_{\mathrm{c}}\right)+{\mathrm{c}}_{\mathrm{y}}\left({\mathrm{r}}_{31}{\mathrm{X}}_{\mathrm{i}}+{\mathrm{r}}_{32}{\mathrm{Y}}_{\mathrm{i}}+{\mathrm{r}}_{33}\mathrm{H}+{\mathrm{Z}}_{\mathrm{c}}\right)\\ {\mathrm{r}}_{31}{\mathrm{X}}_{\mathrm{i}}+{\mathrm{r}}_{32}{\mathrm{Y}}_{\mathrm{i}}+{\mathrm{r}}_{33}\mathrm{H}+{\mathrm{Z}}_{\mathrm{c}}\end{array}\right)$$

(17)

The coordinates of the four corner points of the bulk unloading hopper in the world coordinate system can be obtained by solving the system of equations $({\mathrm{X}}_1,{\mathrm{Y}}_1,\mathrm{H})$,$({\mathrm{X}}_2,{\mathrm{Y}}_2,\mathrm{H})$,$({\mathrm{X}}_3,{\mathrm{Y}}_3,\mathrm{H})$ and $({\mathrm{X}}_4,{\mathrm{Y}}_4,\mathrm{H})$.

3.4.2. Locate the Center Coordinates of the Bulk Unloading Hopper

The world coordinates of the center point of the bulk unloading hopper $\left(X_h,Y_h,H\right)$ are determined using the coordinates of the four corner points of the hopper in the world coordinate system. These coordinates are obtained by applying the Perspective-n-Point (PnP) algorithm.

$${\mathrm{X}}_{\mathrm{h}}={\displaystyle \frac{{\mathrm{X}}_1+{\mathrm{X}}_2+{\mathrm{X}}_3+{\mathrm{X}}_4}{4}}, {\mathrm{Y}}_{\mathrm{h}}={\displaystyle \frac{{\mathrm{Y}}_1+{\mathrm{Y}}_2+{\mathrm{Y}}_3+{\mathrm{Y}}_4}{4}}, {\mathrm{Z}}_{\mathrm{h}}=\mathrm{H}$$

(18)

4. Experiment and Evaluation

4.1. Experimental Environment and Equipment Configuration

The experiment described in this paper was conducted at the Tianjin Port bulk cargo terminal, as shown in Figure 13. Two GPS units were installed on the gantry crane’s elephant trunk. A vertical vision camera was mounted on the maintenance platform at the tip of the elephant trunk. The camera model is LDS-BP0B96 (2.8 mm), and its main parameters are listed in

Table 2. Camera parameters.

Optical zoom	2.8 mm, Fixed Focus
Field of view	Horizontal 101.7 degrees
Aperture	F1.0
Spotlight	F = 2.8 mm

.

Ubuntu 20.04 serves as the underlying operating system for the experimental ecosystem. The equipment configuration used for the training process includes 16 GB RAM, an NVIDIA GTX 3090 GPU, and an Intel i5-12400F CPU. The edge computing device used is a WIPC-707 industrial computer, equipped with a GeForce GTX 1660 graphics card with 6 GB of video memory. The software framework encompasses torch 1.12.1 + cu113 in conjunction with Anaconda. In our experimental setup, we configure the following parameters: an image size of 1280, 300 training rounds, a batch size of 8. We employ the SGD optimizer, and opt not to utilize cache-based data-grabbing methods. This setup ensures consistency and precision in our experimentation process.

4.2. Experimental Data Collection

Due to the lack of publicly available datasets for bulk unloading hopper operations at bulk cargo terminals, it is necessary to create a custom dataset. This involves capturing a large number of photos in the operational scenes of bulk unloading hoppers at the terminal, followed by image preprocessing and annotation. For this purpose, we collected 1800 photos of bulk unloading hoppers in operation under various weather conditions, and annotated only the class of bulk unloading hopper with the label “Hopper”. The images in the dataset were obtained using a vertical camera mounted on the tip of the gantry crane’s elephant trunk, capturing the live operating scene from above. The resolution of the pre-processed images was 1920 × 1080.

Due to the complexity of the loading and unloading environment at the bulk cargo terminal, the acquired image data needed to be expanded to increase the number and spatial diversity of the dataset. This included applying techniques such as horizontal flipping and luminance enhancement to enhance the learning capability of the deep neural network. After expanding the dataset, a total of 3000 images were collected. The dataset was divided into training and testing sets: the training set was used for training the algorithm models, while the testing set was employed to test and analyze the model performance. The split between the training and testing sets was 7:3. Figure 14 illustrates examples of the photos used in the dataset.

The GPS data are recorded in real time during operation, and the sample contains the X, Y and Z positioning coordinates of two GPSs on the elephant trunk.

4.3. Experimental Results

In the comparison experiments during the rough positioning stage, we evaluated the performance of our method against YOLOv5, which was used as the baseline. Both the baseline model and our model were trained on the same training set. Since different training parameters can affect the final results, the training parameters of the baseline model were set mainly according to its recommended default values with only minor adjustments. The results in

Table 3. Comparison of the detection accuracy of the vision-based baseline method and our method on the dataset.

Method	Params (M)	AP (IoU = 0.5, %)	AP (IoU = 0.5:0.9, %)	FPS
YOLOv5n	1.9	99.5	85.3	67
Ours	1.2	98.3	81.9	102

show that our proposed method achieves a substantial increase in detection speed, approximately 102 FPS faster than the YOLOv5 algorithm, in the bulk cargo terminal loading and unloading operation scenario, without a significant loss in detection accuracy. These improvements lay a solid foundation for future implementations where video streams from multiple gantry cranes will be processed simultaneously on a single server.

In practical scenarios where the conditions do not allow for the use of multiple GPUs for parallel processing, a single server GPU’s inference capability can support up to 67 FPS when executing YOLOv5n model inference tasks. In the actual automated bulk cargo terminal operation, multiple gantry crane video streams are processed in real time by a single server, placing a high computational load on it. Although theoretically, a single server can handle two 30 FPS video streams, practical implementation requires consideration of resource competition, data transmission, and load balancing issues.

Given the limited computational resources, optimizing the inference model to enhance the processing capability of a single GPU is crucial for achieving higher inference efficiency. The improved inference model discussed in this paper achieves a processing speed of 102 FPS. This means that the same server can simultaneously handle the real-time inference tasks of three gantry cranes with 30 FPS video streams, while reserving 12% of the GPU capacity to handle unexpected situations.

During the actual deployment at the site, our engineering team further quantized the model to int8, which significantly enhanced the GPU processing limit. Consequently, the system can meet the simultaneous operational inference requirements of 5 to 6 automated gantry cranes. These optimizations not only improve processing efficiency but also ensure the accuracy of operations, enabling real-time inference tasks for multiple video streams.

Take the sample test video of loading and unloading operation process of Tianjin Port bulk cargo terminal as an example to conduct.

In the on-site experiments, we selected three bulk unloading hoppers at different coordinates. Ten camera positions were set based on the heading angle of the grab-type gantry crane to predict the coordinates of the bulk unloading hoppers. Video streams were captured under varying weather conditions and changes in lighting. The positions of the hoppers in the camera images under different heading angles are shown in Figure 15.

By comparing the actual world coordinates of the bulk unloading hoppers with the predicted coordinates and calculating their accuracy at the scale of the bulk unloading hopper, we obtained the results shown in

Table 4. Experiment on positioning accuracy in fully automated loading and unloading operations at bulk cargo terminals.

Serial Number	Actual Coordinates $\mathbf{(}{\mathbf{X}}_{\mathbf{H}},{\mathbf{Y}}_{\mathbf{H}},{\mathbf{Z}}_{\mathbf{H}}\mathbf{)}$	Predicted Coordinates $\mathbf{(}{\mathbf{X}}_{\mathbf{P}},{\mathbf{Y}}_{\mathbf{P}},{\mathbf{Z}}_{\mathbf{P}}\mathbf{)}$	Offset Distance (m)	Prediction Accuracy P (%)
1	(187.900, 82.479, 7.100)	(187.864, 82.418, 7.100)	0.07	98.56
2		(188.183, 82.418, 7.100)	0.29	94.15
3		(187.894, 82.487, 7.100)	0.01	99.80
4		(187.853, 82.486, 7.100)	0.05	99.04
5		(188.031, 82.431, 7.100)	0.14	97.18
6		(187.915, 82.496, 7.100)	0.02	99.55
7		(187.799, 82.303, 7.100)	0.20	95.90
8		(188.191, 82.417, 7.100)	0.30	93.99
9		(187.704, 82.706, 7.100)	0.30	93.95
10		(187.906, 82.475, 7.100)	0.01	99.85
11	(126.217, 81.905, 7.100)	(126.129, 82.072, 7.100)	0.19	96.19
12		(126.050, 81.656, 7.100)	0.30	93.94
13		(126.007, 81.706, 7.100)	0.29	94.16
14		(125.985, 82.091, 7.100)	0.30	93.99
15		(126.316, 81.901, 7.100)	0.10	98.00
16		(126.211, 81.730, 7.100)	0.18	96.46
17		(126.437, 81.763, 7.100)	0.26	94.71
18		(126.282, 81.613, 7.100)	0.30	93.96
19		(125.936, 81.906, 7.100)	0.28	94.32
20		(125.949, 81.787, 7.100)	0.29	94.08
21	(−46.791, 82.857, 7.100)	(−47.004, 82.647, 7.100)	0.30	93.95
22		(−46.795, 82.844, 7.100)	0.01	99.72
23		(−46.667, 82.669, 7.100)	0.23	95.45
24		(−46.892, 83.067, 7.100)	0.23	95.29
25		(−46.584, 83.026, 7.100)	0.27	94.61
26		(−46.726, 82.861, 7.100)	0.06	98.69
27		(−46.585, 82.639, 7.100)	0.30	93.94
28		(−46.774, 82.871, 7.100)	0.02	99.56
29		(−46.546, 83.027, 7.100)	0.30	93.98
30		(−46.625, 82.670, 7.100)	0.25	94.95
Average			0.20	96.06

. The true coordinates of the center of the bulk unloading hopper were determined using RTK measurement, as shown in Figure 16.

During the automated loading and unloading operations at the bulk cargo terminal, we employed this positioning method. We conducted a total of 30 coordinate positioning tests at different coordinates and heading angles, as depicted in Figure 17. The data obtained are summarized in Table 4.

Using this method, the average positioning error of the bulk unloading hopper in the automated operations was 0.20 m, with an average positioning accuracy of 96.06%. This exceeds the technical requirement of 90% alignment accuracy in manual operations, demonstrating that the alignment accuracy of this method has reached and surpassed the level achievable by manual operations. This further validates the effectiveness of the proposed approach.

5. Conclusions

This paper proposes a monocular vision-based, three-stage efficient positioning method for high-precision deviation positioning of bulk unloading hoppers at automated bulk cargo terminals. Accurate measurement of the 3D coordinates of the bulk unloading hopper is achieved using an improved YOLOv5 model and image processing techniques combined with the PnP algorithm. Experimental results show that this method significantly improves the detection speed to 102 FPS without a significant loss of detection accuracy. This detection speed allows a single server to conduct detection for multiple unloading operations simultaneously, thereby reducing the hardware costs of the system.

Furthermore, applying this positioning method to automated loading and unloading operations at the bulk cargo terminal resulted in an average positioning error of 0.20 m over thirty tests, with an average positioning accuracy of 96.06%. This exceeds the 93% technical benchmark of manual operations. These results demonstrate that the method not only improves operational efficiency but also surpasses the accuracy and reliability of manual operations, indicating a wide range of practical application prospects. For the practical deployment in the field, we also performed int8 quantization on the model. This process converts the floating-point weights in the model to 8-bit integers, significantly reducing the model’s size and increasing its inference speed, making it more suitable for field operations.

Indeed, the robustness of the system still requires improvement. The harsh working conditions at bulk cargo terminals, such as dust covering the cameras, can affect the overall stability of the system. Additionally, the swinging motion of the bulk cargo grab is an important factor to consider during unloading operations. In the future, bulk cargo terminals will need to focus on maintaining stable operation, achieving rapid and precise positioning, and integrating the measurement of the bulk grab’s posture with automatic control systems to realize efficient automation of gantry crane unloading operations.