Integrated Scale-Adaptive Adjustment Factor-Enhanced BlendMask Method for Pineapple Processing System

Abstract

This study addresses the challenge of efficiently peeling pineapples, which have a distinct elliptical form, thick skin, and small eyes that are difficult to detect with conventional automated methods. This results in significant flesh waste. To improve the process, we developed an integrated system combining an enhanced BlendMask method, termed SAAF-BlendMask, and a Pose Correction Planning (PCP) method. SAAF-BlendMask improves the detection of small pineapple eyes, while PCP ensures accurate posture adjustment for precise path planning. The system uses 3D vision and deep learning technologies, achieving an average precision (AP) of 73.04% and a small object precision (APs) of 62.54% in eye detection, with a path planning success rate reaching 99%. The fully automated electromechanical system was tested on 110 real pineapples, demonstrating a reduction in flesh waste by 11.7% compared to traditional methods. This study highlights the potential of advanced machine vision and robotics in enhancing the efficiency and precision of food processing.

1. Introduction

In recent years, the global production of pineapples has shown a continuous upward trend. It was estimated that in 2020, approximately 40% of the global pineapple production (around 10 billion pineapples) was utilized for processing, yielding canned pineapples, pineapple juice, and other derivative products [1]. However, during processing, especially when dealing with the pineapple’s distinctive elliptical shape, thick outer skin, and deeply recessed pineapple eyes, the inability to accurately locate the pineapple eyes results in imprecise automated processing. Consequently, pineapples processed by machinery resembling the Ginaca machine [2,3] suffer from pronounced pulp wastage. Cylindrical cutting leads to significant wastage, as illustrated in Figure 1a. Manual handling using specialized tools has emerged as a common approach to address non-standard processing, offering flexibility and processing precision, as depicted in Figure 1b, illustrating the manual processing of pineapples. However, it requires a considerable amount of manual labor. The key aspects of automated pineapple processing include the precise identification of pineapple eyes and their rapid and accurate removal.

Artificial intelligence, especially deep learning, has revolutionized the field of fruit recognition and classification for the past few years. Technologies like Convolutional Neural Networks (CNNs) have been widely used for visual fruit recognition [4,5,6,7,8]. For instance, Zhang et al. [9] developed a five-layer CNN model for fruit classification. This model was applied to both the UEC-FOOD100 dataset [10] and an in-house fruit dataset. Liu et al. [11] utilized a YOLO-v3 [12] single-stage detector, enhancing it to accurately predict circular regions for tomato localization in intricate scenes. Yu et al. [13] introduced a Mask R-CNN [14] algorithm for recognizing 100 wild strawberry images. The results indicated an average recognition accuracy of 95.78% and a recall rate of 95.41%. In 2021, Wang et al. [15] proposed a modified VGG16 [16] network to count apple flowers in images. The total flower count represented the image class, and the network was trained to estimate the visible flower count without localization. Chen et al. [17] proposed another deep learning approach to perform segmentation, rather than detection, using a DCNN.

However, there is a significant scale difference between the pineapple contour and the pineapple eyes, resulting in low detection performance for small target pineapple eyes. Moreover, during the peeling process, excessive removal of the peel results in particularly small retained pineapple eye sizes, exacerbating the difficulty of small target recognition. To address the challenge of small object detection, researchers have proposed various methods, which can be categorized as follows: data augmentation methods to address the scarcity and non-uniform distribution of small object data. Kisanta et al. [18] improved the recognition accuracy and segmentation accuracy of small objects by 7.1% and 9.7%, respectively, through oversampling and augmentation. The Mosaic data augmentation method proposed in YOLOv4 [19] randomly splices four images together by flipping, scaling, and cropping, reducing the computational load on the graphics processing unit (GPU), but potentially decreasing the model’s generalization ability due to smaller object sizes. For super-resolution methods addressing the weak visual features of small objects, Haris et al. [20] proposed an end-to-end super-resolution method that uses the Faster R-CNN network to process low-resolution regions, improving small object detection performance. Bai et al. [21] proposed the multitask generative adversarial network (MTGAN), which, based on super-resolution methods, can up-sample blurry images to generate fine images corresponding to the desired scale for more accurate detection. For multi-scale fusion methods addressing the weak representation capability of single feature layers for small objects, Lin et al. [22] proposed feature pyramid networks (FPN), which integrate low-resolution strong semantic features with high-resolution weak semantic features to enhance the information of feature maps, fully exploiting the semantic information of each different scale feature map, thereby providing certain assistance for small object detection. However, during the fusion process, there may be issues with semantic gaps and noise, which to some extent hinder the further improvement of small object detection methods in terms of performance.

Pineapple processing has also gone through different stages. Traditional pineapple processing methods involve manual operations that require skilled labor to use various tools for tasks such as peeling, coring, and removing eyes. While manual peeling and slicing tools, as described in references [23,24,25] exist, they are challenging to operate and have low efficiency. To address labor challenges, automated processing methods, including mechanical peeling and coring machines, have been introduced. These methods have evolved over time and encompass various technologies. For instance, Hawaiian Pineapple Company (HPC) hired Henry Ginaca in 1911 developing a machine (Ginaca machine) that produced pineapple cylinders at a much higher rate [2]. Ginaca machines boast high efficiency but employ a relatively coarse processing method, resulting in significant fruit loss. Ref. [26] designed a semi-automatic peeling principle machine, with a particular focus on the stability of pineapple positioning. However, this machine still employs cylindrical blades, leading to substantial fruit loss during peeling. Ref. [27] designed a semi-automatic peeling principle machine that used a new process, namely rotary peeling, and conducted theoretical analysis along with experimental verification. Although these methods have relatively higher efficiency in peeling, they still suffer from issues like lower overall efficiency, the inability to remove eyes, and the need for manual eye removal, or imprecise processing that results in significant loss of pineapple pulp.

To date, there have been no complete studies on the automatic processing system of pineapple with integrated machine vision recognition. In this study, we propose an integrated approach for pineapple eye removal, incorporating a scale-adaptive adjustment factor and pose correction planning method. Additionally, we design and develop an innovative 3D vision-based intelligent motor-integrated pineapple processing system. This system achieves accurate identification and efficient automation of pineapple fruit processing, providing valuable insights into the automation of agricultural product processing.

The key contributions of this research are as follows:

(1)

We introduce a Scale-Adaptive Adjustment Factor (SAAF) to enhance the effectiveness of the BlendMask target segmentation algorithm, leading to significant improvements in detecting small targets. Multiple experiments have demonstrated that our algorithm exhibits robust performance and strong adaptability across different datasets.

(2)

We propose a Pose Correction Planning Method for the precise cutting path of pineapple eyes, ensuring a high success rate in path planning. This method achieves a path planning success rate of 99%.

(3)

We develop a fully automated, 3D vision-based intelligent motor-integrated pineapple processing system. The proposed algorithms are rigorously validated, demonstrating an average reduction in fruit wastage of 11.7% compared to traditional automated processing methods, such as the Ginaca machine [2].

2. Materials and Methods

As shown in Figure 2, the 3D vision-based mechatronics integrated pineapple processing system consists of the pineapple clamping platform mechanism, a peeling mechanism (which we constructed by referencing these work [25,28]), a 3D vision system, and an eye removal mechanism. The specific steps are as follows.

(1)

Data collecting—employing RGB-D camera to uniformly capture six images of each peeled pineapple, comprising both RGB imagery and depth information;

(2)

Image processing—utilizing an improved BlendMask algorithm (SAAF-BlendMask) for the identification of pineapples and pineapple eyes, followed by the reconstruction of the point cloud distribution of pineapple eyes;

(3)

Motion planning—planning of cutting paths (PCP method) and trajectories based on the distribution of pineapple eyes and cutting the pineapple eyes following the planned trajectories.

In particular, precise removal of pineapple eyes is critical and is achieved by emulating the manual removal process, particularly in steps (2) and (3). Accurate identification of pineapple eyes and the planning of removing trajectories are of paramount importance. The premise for trajectory planning is the alignment and maintenance of the pineapple’s central axis with the rotational axis, which is achieved in step (3). The following elaborate exposition will be presented.

2.1. Data Collecting

The pineapple needs to be peeled prior to image acquisition, as depicted in Figure 2. We have designed a peeling and clamping mechanism for this purpose. During the peeling process, the knife follows a spiral pattern on the pineapple’s surface, effectively removing the peel. This peeling mechanism maintains a relatively consistent peel thickness, typically ranging between 6 and 9 mm. This minimizes fruit wastage while retaining the pineapple eyes.

We opted for an RGB-D camera, specifically the Intel RealSense D435, to establish a depth network vision system. An RGB-D camera captures RGB images with depth values, or Z-depth, which are transformed into a 3D point cloud in the camera coordinate system $O_c$ using intrinsic parameters, and further converted to a global coordinate system $O_w$ using extrinsic parameters, as shown in Figure 3b.

Figure 3a provides a visual representation of the camera’s field of view. It is noteworthy that points at greater distances from $O_c$ exhibit higher errors (larger physical sizes corresponding to the same pixel), while points too close may become out of focus. Establishing an acceptable range is crucial to ensuring the uniform and accurate capture of points by the camera, as depicted in Figure 3a. Therefore, during camera installation, we aim to position the camera as close to the pineapple as possible within the effective detection range. This is to ensure that the pineapple eyes occupy a greater number of pixels.

Pineapple Image Acquisition. To enhance the effectiveness and robustness of our algorithm, we collected pineapple images from real-world scenarios.

Prior to image acquisition, the visual system was installed and calibrated with the processing equipment according to the technical specifications of the D435 camera (Intel Corporation, Santa Clara, CA, USA).

In order to reconstruct the complete point cloud distribution of pineapple eyes more accurately, multiple images of each pineapple were captured from different angles. Each image had a resolution of 640 × 480 pixels, and images were captured at 60-degree intervals. After every capture, the pineapple was rotated by 60 degrees. After six captures, the pineapple returned to its initial angle, ensuring consistency between the reconstructed pineapple pose and its actual pose.

Then, a significant number of images were collected and processed using data augmentation techniques, such as brightness adjustments, to account for varying lighting conditions in the processing environment. The collected images were further processed using the Labelme annotation software (version is 4.5.9, created by Kentaro Wada, and sourced from Berkeley, CA, USA).

2.2. SAAF-BlendMask Algorithm

The BlendMask algorithm belongs to one-stage dense instance segmentation methods. It is based on the anchor-free detection model FCOS [29] and combines instance-level information (represented by the orange dashed box in Figure 4) and finer-grained semantic information (represented by the green dashed box in Figure 4) to enhance mask prediction. It also introduces a blending module (represented by the yellow-green block in Figure 4), combining top-down and bottom-up instance segmentation methods. This effectively predicts position-sensitive instance features for dense pixel points using only one convolutional layer, preserving advantages in speed. In some cases, it outperforms Mask R-CNN in both speed and performance, meeting the real-time processing requirements of the system [30].

During reconstruction, each pineapple is reconstructed from the point cloud extracted from six images, with overlapping point clouds between adjacent images. Therefore, precision in point cloud extraction is particularly crucial. In the task of pineapple eye recognition and segmentation, segmentation networks can more accurately extract point cloud information relevant to pineapple eyes compared to traditional object detection methods (which typically use bounding boxes to label targets). Hence, we introduce the real-time instance segmentation algorithm BlendMask as the baseline.

To enhance the recognition performance of small-scale pineapple eyes and mitigate the influence of pineapple contour variations and eye size discrepancies, we devised a scale-adaptive balancing factor to refine the BlendMask algorithm [30]. The overall architecture of the vision system algorithm, as depicted in Figure 4, comprises two main components: instance segmentation and 3D point cloud reconstruction.

This algorithm employs a pyramid structure network (FPN), efficiently extracting feature maps at different resolutions, addressing the issue of target overlap. To address the challenge posed by the imbalance in size between pineapples and pineapple eyes in this application scenario, we proposed the Scale-Adaptive Adjustment Factor (SAAF) to enhance the BlendMask loss function, thereby improving the recognition accuracy of small target pineapple eyes.

In the FCOS architecture’s head module, pixel-wise target detection is performed using different hierarchical feature maps to predict targets, regressing 4D vectors $T^{\ast }=(l^{\ast },b^{\ast },r^{\ast },t^{\ast })$ for the target bounding boxes to improve accuracy, as shown in Figure 5. To enhance accuracy further, the concept of centerness [29] is introduced to eliminate low-quality predicted boxes that are far from the center.

$$centernes{s}^{\ast }=\sqrt{{\displaystyle \frac{min(l^{\ast },r^{\ast })}{max(l^{\ast },r^{\ast })}}\ast {\displaystyle \frac{min(t^{\ast },b^{\ast })}{max(t^{\ast },b^{\ast })}}}.$$

(1)

To better address the issue of large-scale differences between pineapple eyes and pineapples leading to overlap, we conducted a detailed statistical analysis. The results indicated that the average resolution of pineapple eyes is about 20 × 20 pixels, while pineapples have an average resolution of about 200 × 250 pixels. Therefore, we adapted FCOS’s feature maps with five different sizes, denoted as $P3$, $P4$, $P5$, $P6$ and $P7$, each with corresponding strides of 8, 16, 32, 64, and 128. We limited the regression distances differently for each feature map to achieve this purpose. Specifically, each feature map $Pi$ was assigned lower and upper bounds $m_{i-1}$ and mi as constraints for regression distances. For each coordinate point $(x,y)$ on feature map $Pi$, if $max(l,t,r,b)>m_i$ or $min(l,t,r,b)<m_{i-1}$, it was treated as a negative sample on that feature layer, and regression was no longer performed. $m2$, $m3$, $m4$, $m5$, $m6$ are set to 0, 64, 128, 256, and +∞, respectively. This implies that feature map $P3$ handles target regression values within the [0, 64] range, while feature map $P7$ is responsible for targets with regression values greater than 256.

In the previous analysis, although FCOS employs hierarchical feature maps to predict targets of different scales, the practical situation is that, on average, the number of anchor points in pineapple contour significantly exceed the number of anchor points on pineapple eyes. This imbalance leads to differences in centerness weight between anchor points on large pineapple targets and small pineapple eyes. Specifically, the centerness weight of small pineapple eyes tends to be lower, thus, they are often treated as low-quality regression boxes and suppressed. However, in the processing pipeline of this study, the peeling operation results in significant differences in the size of retained pineapple eyes, including very small pineapple eyes being quite common. Since the system’s goal is to accurately remove each residual pineapple eye, identifying these commonly occurring small pineapple eyes becomes crucial. To address the issue of weight imbalance caused by differences in target scales, we introduced the SAAF module to enhance the centerness. This enhancement aims to ensure that small pineapple eyes are equally well-recognized and processed, thereby improving the overall performance of the system.

The inputs of the SAAF module consist of ground truth and all positive samples, while the output is the adjustment factor. Initially, we compute the average area ${\mu }_{sgt}$ of the ground truth,

$${\mu }_{sgt}=\sum _i^{N_{gt}}{s}_{gti}/N_{gt},$$

(2)

where $s_{gti}$ represents the area of the $i-th$ ground truth, $N_{gt}$ represents the number of ground truths. corresponding to the area $s_{sa(x,y)}$ of each positive sample (at the $(x,y)$ point) from the ground truth.

$$s_{sa(x,y)}=(l^{\ast }+r^{\ast })\ast (b^{\ast }+t^{\ast }).$$

(3)

We determine the centerness of each positive sample and calculate a factor ${\omega }_{s(x,y)}$ for each sample’s centerness.

$${\omega }_{s(x,y)}=\sqrt{{\displaystyle \frac{{\mu }_{sgt}}{s_{sa(x,y)}}}},$$

(4)

$${\mu }_{\omega s(x,y)}=\sum _{x,y}{\omega }_{s(x,y)}/N_s,$$

(5)

this is achieved by comparing against a threshold ${\mu }_{\omega s(x,y)}$, where $N_s$ represents the number of positive samples for each ground truth.

$$k_{x,y}=\left\{\begin{array}{cc}\hfill & {\omega }_{s(x,y)}\ast m,({\omega }_{s(x,y)}>{\mu }_{\omega s(x,y)})\hfill \\ \hfill & 1\ast n,({\omega }_{s(x,y)}<{\mu }_{\omega s(x,y)})\hfill \end{array}\right.,$$

(6)

where m and n represent hyperparameters, with a default value of 1. Therefore, each positive sample’s weight is represented as $s_{x,y}=k_{x,y}\ast centernes{s}^{\ast }$, which means it is adaptively adjusted based on the ground truth scale. After this adjustment, the loss function still consists of three parts: classification loss, regression loss, and centerness loss, combined.

$$\begin{array}{cc}\hfill & L(\left\{p_{x,y}\right\},\left\{t_{x,y}\right\},\left\{s_{x,y}\right\})\hfill \\ \hfill =& {\displaystyle \frac{1}{N_s}}\sum _{x,y}{L}_{cls}(p_{x,y},c_{x,y}^{\ast })\hfill \\ \hfill & +{\displaystyle \frac{1}{N_s}}\sum _{x,y}{1}_{\{s_{x,y}^{\ast }>0,c_{x,y}^{\ast }>0\}}{L}_{reg}(t_{x,y},t_{x,y}^{\ast })\hfill \\ \hfill & +{\displaystyle \frac{1}{N_s}}\sum _{x,y}{1}_{\{c_{x,y}^{\ast }>0\}}{L}_{ctr}(s_{x,y},s_{x,y}^{\ast }),\hfill \end{array}$$

(7)

where $p_{x,y}$ represents the scores for each class predicted at the $(x,y)$ point in the feature map. $t_{x,y}$ represents the predicted target bounding box information at the $(x,y)$ point in the feature map. $s_{x,y}$ represents the centerness weight predicted at the $(x,y)$ point in the feature map. $N_s$ represents the number of positive samples. $c_{x,y}^{\ast }$ represents the true class labels at the $(x,y)$ point in the feature map. $1_{s_{x,y}^{\ast }>0}$ is 1 when the $(x,y)$ point in the feature map is matched as a positive sample, otherwise 0. $1_{c_{x,y}^{\ast }>0}$ is 1 when the adjusted weight at the $(x,y)$ point is greater than 0, otherwise 0. $t_{x,y}^{\ast }$ represents the true target bounding box information corresponding to the $(x,y)$ point in the feature map. $s_{x,y}^{\ast }$ represents the weight-adjusted adaptively through the $(x,y)$ point in the feature map.

Next, we transformed the depth images into point clouds and integrated them with segmented pineapple eye regions to generate corresponding pineapple eye point cloud data. We successfully stitched together point clouds for all pineapple eyes, as shown in Figure 4, achieving a high-quality reconstruction of the entire pineapple. To achieve the effect of artificially removing pineapple eyes, as shown in Figure 1b, the next step is planning the cutting path.

Next, we transformed the depth images into point clouds and integrated them with segmented pineapple eye regions to generate corresponding pineapple eye point cloud data. To facilitate understanding and computation, we establish the origin $O_w$ of the world coordinate system on the rotation axis of the pineapple, with the Z-axis aligned along the rotation axis and the Y-axis oriented vertically upward, as illustrated in Figure 2. Assuming the camera’s extrinsic matrix is denoted as ${}^{w}T_c$, and the point cloud obtained from image segmentation after each capture is represented as $P_{ci}$ for $i\in \{1,\dots ,6\}$. The world coordinate system’s point cloud $P_{wi}$ is given by:

$$P_{wi}{=}^w{T}_c{P}_{ci}.$$

(8)

After each capture, the pineapple undergoes a 60° rotation around the Z-axis, with the corresponding rotation transformation matrix denoted as $T_{60}$. Consequently, the previously obtained world coordinate system’s point cloud $P_{wi}$ undergoes a rotation and is updated accordingly:

$$P_{wi}=T_{w60}{P}_{wi}.$$

(9)

This rotation operation is performed each time a new set of data is collected.

Through the above operations, we successfully stitched together point clouds for all pineapple eyes, as shown in Figure 4, achieving a high-quality reconstruction of the entire pineapple. However, to achieve the effect of artificially removing pineapple eyes, as shown in Figure 1b, cutting-path planning is necessary.

2.3. Pose Correction Planning (PCP) Method

After reviewing the literature [31,32], we found that the distribution of pineapple eyes on each pineapple follows 8 helical lines. Using a specialized tool [23], we can cut along each helical line, thereby removing all pineapple eyes after cutting along the 8 helical lines.

However, during the pineapple loading process, it cannot be ensured that the rotation axis of the pineapple aligns with its own central axis. As illustrated in Figure 6, the red dashed line represents the rotational axis of the pineapple (i.e., the Z-axis of the mechanism coordinate system constructed in Section 3.1), while the green dashed line denotes the central axis of the pineapple. The challenges and focal points lie in how a computer can accurately identify the 8 helical lines of pineapple eyes and quickly and precisely remove them. To address these issues, we developed the Pose Correction Planning (PCP) method, as outlined in Algorithm 1. The key steps are as follows:

(1)

Uniform sampling of the pineapple surface point cloud ($S_p$) yields a consistent and well-distributed point cloud ($S_{ps}$) of the pineapple surface.

(2)

Use Principal Component Analysis ($myPCA(·)$) to determine the pose or coordinate system ($O_p$) of the pineapple.

(3)

Compute the centroid ($P_{wi}$) of the pineapple eye surface point cloud obtained with previous sections of the paper.

(4)

Calculate the coordinates ($P_{pi}$) of the point cloud centroid in the pineapple coordinate system.

(5)

The distribution of point cloud centroids ($P_{pi}$) is transformed from the pineapple coordinate system to the radian-height coordinate system ($C_{pi(\theta -z)}$), where the radian is the horizontal axis ($\theta -axis$) and the $z-axis$ is the vertical axis, as shown in Figure 7. It can be seen that the distribution of the pineapple eyes in this coordinate system is quite regular, with the pineapple eyes on the same spiral line essentially aligned in a straight line. Therefore, we thought of using clustering to distinguish the pineapple eyes on each spiral line.

(6)

Since the distribution radian of the pineapple eyes ranges from 0 to $2\pi$, we first need to shift the pineapple eyes on the same spiral line before clustering. Assuming the slope of the straight line formed by the distribution of pineapple eyes is k, and the original coordinates of pineapple eyes are $C_{pi}({\theta }_i,z_i)$, the shifted coordinates $C_i^{\prime }({\theta }_i^{\prime },z_i^{\prime })$ are:

$${\theta }_i^{\prime }=\left\{\begin{array}{cc}{\theta }_i-2\pi \hfill & \mathrm{if}{\theta }_i-{\displaystyle \frac{z_i-min\left(z_i\right)}{k}}>2\pi \hfill \\ {\theta }_i+2\pi \hfill & \mathrm{if}{\theta }_i-{\displaystyle \frac{z_i-min\left(z_i\right)}{k}}<0\hfill \\ {\theta }_i\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(10)

Then, we rotate the obtained $C_i^{\prime }$ by $\alpha$ and calculate the $cost$ value, and $\alpha$ ranges from ${\displaystyle \frac{\pi }{6}}$ to ${\displaystyle \frac{\pi }{3}}$. The relationship between $\alpha$ and k is as follows:

$$k=-1.0·tan\left(\alpha \right).$$

(11)

The coordinates after rotation, $N{C}_i^{\prime }$, are:

$${NC}_i^{\prime }=\left[\begin{array}{cc}cos\left(\alpha \right)& -sin\left(\alpha \right)\\ sin\left(\alpha \right)& cos\left(\alpha \right)\end{array}\right]·C_i^{\prime }.$$

(12)

With each rotation, we perform clustering ($MyClusterAL(·)$) to obtain 8 groups matrices ${NC}_j^{\prime }$ ($j\in \{1,\dots ,8\}$). Each element ${NC}_{ij}^{\prime }({\theta }_{ij},z_{ij})$ represents a pineapple eye, where j is the index of the spiral line (ranging from 1 to 8), and i is the number of pineapple eyes on each spiral line (ranging from 1 to $Nu{m}_j$). If ${\theta }_{ij}=0$, then $N{C}_{ij}^{\prime }({\theta }_{ij},z_{ij})$ can be represented as $N{C}_{ij}^{\prime }\left(z_{ij}\right)$. The cost function ($L_{cost}$) is designed to evaluate the quality of a given scheme by penalizing the variance of different variables. Specifically:

$$\begin{array}{c}{L}_{cost}=k_{1}Var\left(Nu{m}_j\right)+k_2{\displaystyle \frac{1}{8}}\sum _{j=1}^{8}Var\left(\parallel {NC}_{ij}^{\prime }\left(z\right)-{\overline{{NC}^{\prime }}}_j\left(z\right)\parallel \right)\hfill \\ \hfill +k_3{\displaystyle \frac{1}{8}}\sum _{j=1}^{8}Var\left(\parallel {NC}_{[i+1]j}^{\prime }-{NC}_{\left[i\right]j}^{\prime }\parallel \right),(i\in \left(1,2,\dots ,Nu{m}_j\right),j\in \left(1,2,\dots ,8\right)),\end{array}$$

(13)

where $k_1$, $k_2$ and $k_3$ are hyperparameters that adjust the weight of each term in the overall cost function. $Var(·)$ represents the variance calculation. The calculation of ${\overline{{NC}^{\prime }}}_j\left(z\right)$ is as follows:

$${\overline{{NC}^{\prime }}}_j\left(z\right)={\displaystyle \frac{1}{Nu{m}_j}}\sum _{i=1}^{Nu{m}_j}{{NC}^{\prime }}_{ij}\left(z\right).$$

(14)

(7)

In the mechanism coordinate system, sort the pineapple eyes on each helical line according to their Z-axis values to obtain an ordered set of pineapple eyes for each helical line. Connect the pineapple eyes on each helical line with line segments.

Algorithm 1: PCP (Pose Correction Planning) Method

To enhance processing efficiency, we also designed a cooperative dual-blade eye removal mechanism tailored to the specific characteristics, as illustrated in Figure 8. This mechanism comprises symmetric left and right eye-cutting devices, each equipped with three degrees of freedom, including the eye cutter, eye cutter rotary motor, eye cutter feed motor, eye cutter translation motor, and other essential components. Collaborating with the clamping platform’s pineapple rotation axis, they simultaneously cut two lines, repeat the process twice, and successfully removed the pineapple eyes along all eight spirals.

3. Results and Discussion

To validate the effectiveness of the automated processing system, we constructed a prototype of the pineapple automated processing system, as shown in Figure 8. It is comprised of an automatic feeding mechanism (including conveyor feeding and pose correction), automatic loading mechanism, peeling mechanism, RGB-D camera (D435), eye removal mechanism, and a computer (operating system: Ubuntu 20.04). This computer is equipped with 64 GB of RAM, an Intel Core i9-10940X processor, and an NVIDIA GTX3090 graphics processor. The camera has an RGB resolution of 640 × 480 pixels and operates at a frame rate of 30 fps. Different modules work in a predefined sequence, enabling the entire automated processing process. In the experiments, we conducted separate tests and statistical analyses of each module and then verified the feasibility and stability of the entire automated processing system.

3.1. Result Analysis of Pineapple Eyes Detection

In the system we constructed, the camera was installed directly above the pineapple at a distance of 35 mm from the rotation axis and calibrated with the mechanical setup. The computer is responsible for orchestrating the system’s overall control, which encompasses the regulation of individual components or subsystems, the implementation of deep vision algorithms, as well as the deployment of cutting path and trajectory planning algorithms. This visual system is a core component of our automated processing system and is responsible for identifying the position and shape of pineapple eyes.

In the actual processing environment, we processed 100 pineapples using the automatic peeling mechanism, resulting in a total of 600 pineapple images collected under varying lighting conditions. To increase data diversity and enhance the robustness of the deep vision network, we applied data augmentation techniques. Ultimately, we obtained a PineappleEyes dataset comprising 3000 pineapple images. We used the Labelme software to perform precise annotations of pineapples and pineapple eyes. The dataset was divided into training, validation, and testing sets in an 8:1:1 ratio, which were used for training, validating, and testing the deep vision network and point cloud reconstruction.

In the experiments, we compared the performance of various deep learning models, including YOLACT, SOLOv2, Mask-RCNN, BlendMask, and our enhanced SAAF-BlendMask algorithm, using the pineapple dataset established earlier. Additionally, extensive experiments were conducted to explore different values of the hyperparameters m and n. The experimental results, as depicted in

Table 1. Evaluation results for segmentation (SEGM) with SAAF-BlendMask.

m	n	AP	AP50	AP75	APs	APl	Infer Time (ms)
1	20	72.34	94.17	82.31	62.05	83.19	0.020311
10	20	72.33	94.48	80.68	60.72	83.94	0.020599
20	20	74.27	94.95	82.47	62.44	86.11	0.020481
30	20	73.37	94.96	82.71	61.82	84.92	0.01985
1	30	73.04	95.44	83.62	62.54	83.56	0.020537
10	30	72.10	92.25	82.87	62.35	82.25	0.020342
20	30	70.84	92.78	81.74	62.69	79.00	0.020744
30	30	70.57	91.86	81.29	62.56	78.24	0.020971

and

Table 2. Evaluation results for segmentation (SEGM) with different methods on 8:1:1 PineappleEyes Dataset Split.

Methods	AP	AP50	AP75	APs	APl	Infer Time (ms)
YOLACT	72.89	94.02	82.78	60.42	84.51	0.014947
SOLOv2	72.11	93.17	80.86	58.44	84.72	0.014527
Mask-RCNN	72.85	92.69	81.77	62.29	83.42	0.053034
BlendMask	70.26	94.41	77.59	56.55	83.96	0.019867
SAAF-BlendMask (Ours)	73.04	95.44	83.62	62.54	83.56	0.020537

, highlight the exceptional performance of our enhanced algorithm. The detection results for a single instance of pineapple eye recognition are shown in Figure 9.

To further analyze the impact of the PineappleEyes dataset splitting on model stability, we conducted additional experiments with a different data split ratio of 6:2:2 (training:validation:testing). These experiments were performed using the same models: YOLACT, SOLOv2, Mask-RCNN, BlendMask, and SAAF-BlendMask. As presented in

Table 3. Evaluation results for segmentation (SEGM) with different methods on 6:2:2 PineappleEyes Dataset Split.

Methods	AP	AP50	AP75	APs	APl	Infer Time (ms)
YOLACT	70.86	97.46	69.00	46.95	94.78	0.014475
SOLOv2	69.46	93.36	67.74	42.60	97.33	0.015145
Mask-RCNN	70.96	98.70	79.40	54.25	86.78	0.054751
BlendMask	67.05	97.07	65.50	45.92	91.18	0.019354
SAAF-BlendMask (Ours)	74.70	98.65	76.93	54.78	95.61	0.020245

, the results demonstrate that the SAAF-BlendMask algorithm maintains its superior performance across different dataset splits. Specifically, under the 6:2:2 split, SAAF-BlendMask continued to outperform other algorithms in terms of both average precision (AP) and small object precision (APs).

It is evident that the inference time of the BlendMask algorithm and our improved BlendMask algorithm are closely comparable, significantly outperforming the Mask-RCNN algorithm by approximately 62% in inference time. Moreover, compared to the Mask-RCNN algorithm, our enhanced BlendMask algorithm demonstrated a higher AP50. As shown in Table 2, when $m=1$ and $n=30$, the $AP$ for Mask-RCNN improved by nearly 0.2 percentage points, and for BlendMask it improved by nearly 2.78 percentage points, reaching 73.04%. For the recognition metric focused on small objects, $APs$, the enhanced BlendMask algorithm outperformed BlendMask by nearly 6 percentage points, reaching 62.54%. Under the 6:2:2 split, as presented in Table 3, the trends were consistent with those observed in the 8:1:1 split. The $AP$ was also nearly 7.6 percentage points higher than BlendMask, reaching 74.70%, while the APs improved by almost 9 percentage points, reaching 54.78%, further reinforcing the robustness and adaptability of the proposed SAAF-BlendMask algorithm.

In addition, to further validate the performance and interpretability of the SAAF-BlendMask algorithm, we conducted an additional set of experiments using a subset of the GraspNet-1Billion dataset [33]. This dataset is widely recognized for containing small objects, making it an appropriate benchmark to assess the algorithm’s generalization capability beyond the specific domain of pineapple processing. Given the substantial size of the GraspNet-1Billion dataset, we selected one image from each of 192 different scenes to create a smaller dataset for our testing. This approach allowed us to efficiently evaluate the algorithm while ensuring that the subset was representative of the dataset’s overall diversity. The results of this comparative study are presented in

Table 4. Evaluation results for segmentation (SEGM) with different methods on the GraspNet-1Biliion Dataset.

Methods	AP	AP50	AP75	APs	APl	Infer Time (ms)
YOLACT	73.82	87.39	79.06	24.19	74.52	0.014647
SOLOv2	85.01	92.38	91.53	25.92	89.16	0.015024
Mask-RCNN	78.83	95.21	87.51	18.52	80.69	0.056247
BlendMask	85.24	92.38	91.76	22.13	87.81	0.019875
SAAF-BlendMask(Ours)	85.53	92.50	89.57	29.34	88.02	0.020247

. The SAAF-BlendMask algorithm achieved the highest AP score of 85.53%, outperforming other methods across several metrics, particularly in detecting small objects (APs) with a score of 29.34%.

These results indicate that the SAAF-BlendMask algorithm is not only effective in the specific context of pineapple processing but also demonstrates strong generalization capabilities when applied to external datasets with different characteristics. Despite using a smaller subset of the GraspNet-1Billion dataset, the algorithm’s performance remained robust, further validating its potential for broader applications.

Following pineapple eye segmentation on the RGB images and alignment with the depth images, we extract the corresponding pineapple eye regions from the depth images and convert them into point clouds. During the processing workflow, we capture six images uniformly for each pineapple. We repeat the aforementioned procedure, performing point cloud fusion, denoising, and filtering operations to reconstruct a complete pineapple eye point cloud in the equipment coordinate system, as depicted in Figure 4.

3.2. Result Analysis of Pineapple Eyes Cutting Path Planning

To validate the effectiveness of the PCP method, we conducted a comparative experiment using 100 pineapples. After peeling the pineapples with a machine and reconstructing the pineapple eye point clouds, we planned the cutting path. Initially, the PCP method was not applied, and the results, as shown in Figure 10a,b, depict the distribution of pineapple eyes in the machine coordinate system. Subsequently, the same pineapples underwent cutting path planning using the PCP method, with the pineapple eyes distributed along eight helical lines in the pineapple’s coordinate system, as illustrated in Figure 10c,d. It is clear that the path planned using the PCP method perfectly forms eight spiral lines, achieving the desired effect. This method is more suitable for dual-blade coordinated cutting. The results clearly indicate that path planning with the PCP method is more reliable. The experimental outcomes reveal that 75 pineapples were successfully planned without the PCP method, whereas 99 pineapples were successfully planned with the PCP method.

Following the distribution of pineapple eyes along each spiral, we proceed to plan the cutting trajectories. As outlined in the design of the eye removal apparatus in the preceding sections, we understand that during eye removal, the left and right symmetric eye removal mechanisms collectively exhibit six axes that coordinate with the pineapple’s rotation axis. Using the pineapple rotation axis as the reference, we employ cubic spline interpolation three times for the remaining six axes to obtain the motion trajectories or interpolation data for the eye removal.

During the eye removal process, the computer dynamically controls the synchronized operation of the symmetrical eye removal mechanisms in real-time based on interpolation data, achieving pineapple eye removal (as shown in Figure 11) in a manner that emulates human dexterity (as shown in Figure 1b). In the experiments, we selected 110 pineapples for cutting, divided into four groups based on weight. Initially, we conducted a cutting experiment using our method. After weighing, the peeled pineapples were further processed to achieve the same peeling and eye removal results as traditional methods. The experimental data are presented in

Table 5. Data on pineapple cutting and peeling with different methods.

Weight Group (kg)	<1.2	1.2–1.7	1.7–2.2	>2.2	Total
Number of Pineapples	22	62	19	7	110
Total Weight (kg)	22.65	92.51	37.41	17.16	169.73
Peeling Waste with Our Method (kg)	5.68	22.34	8.79	3.97	40.78
Peeling Waste with Traditional Automatic Processing Method [3] (kg)	6.34	24.92	10.14	4.78	46.18

This table shows the data on pineapple cutting and peeling using different methods. The waste data are calculated based on different peeling techniques.

. Our method of pineapple peeling has demonstrated a significant advantage over the traditional method in terms of minimizing waste. Specifically, the total peeling waste using our method was 40.78 kg, compared to 46.18 kg with traditional automatic processing methods such as those in [3]. The processing effects are similar to those shown in Figure 1a. This indicates that our method preserved approximately 5.40 kg more pineapple flesh, representing a reduction in waste of about 11.7%.

While automatic processing systems can process up to 75 pineapples per minute [3], our approach focuses on optimizing the balance between speed and waste reduction. Traditional manual methods require around 2 min per pineapple [2], but our method achieves the same precision in just 30 s. This balance of efficiency and precision demonstrates the strength of our system in applications where minimizing waste is a priority, even though it may not match the speed of fully automated systems.

3.3. Comparison of Different Pineapple Processing Methods

We also conducted extensive research in the specific domain of automated pineapple processing.

Table 6. Comparative analysis of the proposed framework with the most mainstream techniques.

Methods	IS	LLC	LFW	EP	AIM	EM	Cost
Full Manual Processing [23,25]	×	×	×	×	×	✓	HLC, LEC
Automatic Processing [3]	✓	✓	×	✓	×	✓	HEC
Semi-auto Peeling [26]	×	×	×	✓	×	×	MEC, MLC
Spiral Peeling Processing [27]	×	×	✓	×	×	×	MEC, MLC
Visual Recognition Pineapple Eyes [34]	×	×	×	×	✓	×	MEC, HLC
Ours	✓	✓	✓	✓	✓	✓	MEC

“✓” indicates that the method possesses the corresponding attribute, while “×” indicates that the method does not. Abbreviations: IS (Integrated System), LLC (Low Labor Costs), LFW (Low FleshWastage), EP (Efficient Processing), AIM (AI-based Method), EM (Ensemble Methods), HLC (High Labor Costs), HEC (High Equipment Costs), MEC (Moderate Equipment Costs), MLC (Moderate Labor Costs), LEC (Low Equipment Costs).

compares some of the most popular reported processing methods. Clearly, through comparison, the proposed approach demonstrates more advantages than all methods. The column Integrated System addresses the question of whether a fully automated electromechanical integrated system is provided to address this issue. This paper also compares the use of machine learning model integration. It can be inferred that the proposed system not only fairly addresses the initial problem but also leverages and promotes the use of cutting-edge technology to ensure its relevance in the near future.

4. Conclusions

The developed algorithms, particularly the Scale-Adaptive Adjustment Factor-enhanced BlendMask algorithm and the PCP method, have shown exceptional effectiveness in tackling the challenges of precision pineapple processing. Rigorous experimentation and analysis have demonstrated that these algorithms outperform conventional methods, especially in the areas of pineapple eye detection and path planning accuracy.

The integration of these advanced algorithms into an automated processing system marks a significant advancement in pineapple processing technology. The system, equipped with 3D vision and deep learning capabilities, offers a robust solution for enhancing processing efficiency and reducing fruit wastage in industrial applications.

While this research primarily focuses on the technical aspects of pineapple processing, such as automation and efficiency, it is important to note that our system is designed with minimal human intervention, which inherently enhances food safety and processing consistency. However, the study did not delve deeply into post-harvest quality attributes such as color, texture, tissue damage, or sugar content. These factors are crucial in evaluating the overall quality of processed pineapples, especially for fresh consumption. Our processing method is mainly intended for canned pineapple production, where the primary concern is maximizing yield and minimizing waste rather than optimizing fresh quality attributes. Future work could explore these quality factors in greater detail to provide a more comprehensive comparison between traditional and automated processing methods.

The experimental findings affirm the system’s effectiveness, emphasizing its robustness, reliability, and adaptability in practical scenarios. By optimizing processing workflows and minimizing waste, the system not only boosts productivity but also contributes to resource conservation and sustainability within the food industry.

In summary, the combination of these innovative algorithms and advanced system development represents a transformative approach to pineapple processing. By leveraging cutting-edge technologies, this research sets the stage for more efficient, sustainable, and economically viable agricultural product processing systems.