Fruit Phantoms for Robotic Harvesting Trials—Mango Example

Abstract

Experimental trials on the performance of end-effectors for the automated harvest of soft fruit are constrained by seasonal limitations on fruit availability and fruit perishability, necessitating the use of different sets of fruit across time. Consequently, the use of fruit and stalk phantoms, rather than real fruit, is an attractive proposition. In this paper, a process for the cost-effective production of stable fruit phantoms using silicone (polydimethylsiloxane, PDMS) and starch was presented. A preliminary consideration was also presented for the creation of a phantom fruit stalk, involving a wooden dowel or a magnetic latching. Mango fruit phantoms were benchmarked to mango fruit in terms of density, firmness, brittleness, etc.

1. Introduction

1.1. Requirement

The development of equipment is informed by the comparative assessment of successive versions, which requires use of the same test material. The development of a robotic end effector for the harvest of soft fruit is hampered by the perishability of fruit, such that unless all gripper variants are ready for comparison at the same time, different fruit will be used in assessments. Seasonal limitation on fruit availability is another constraint. The creation of a non-perishable model fruit is therefore desirable to facilitate such assessments.

The harvest of a mango fruit involves a gripping action followed by a rapid twisting action to break the stalk (involving a shear force). The success of these actions will be affected by fruit shape, weight, texture, and firmness (which will impact on the fruit centre of mass and the coefficient of friction between gripper and fruit) and by the breaking force required to detach the mango fruit from its stalk. A model fruit (a ‘phantom’) for the testing of an end effector for fruit harvest should therefore mimic the shape, weight, texture, and firmness of the fruit, and the attachment strength of the stalk. Fruit colour is not important for the gripper action, but is relevant to the overall harvester performance, in terms of fruit detection by machine vision.

1.2. Fruit Phantoms

Wax fruit models were common in the 19th century for economic botany collections and regional produce exhibitions [1]. Hollow plastic (typically low-density polyethylene) and polystyrene fruit are now commonly used for aesthetic and economic botany teaching purposes (e.g., [2]). These models mimic fruit shape and colour, but not weight or texture. Silicone moulds are produced for the production of fruit-shaped soap and edible items (e.g., [3]). Three-dimensional shape files are available for a range of fruit [4] and Turboquid provides models for use in augmented reality [5].

Surprisingly, however, few references to the development of model fruit for use in harvesting research could be found using the keywords of “fruit” and (“model” or “phantom” or “artificial”) and (“harvest” or “postharvest”) in Scopus and Google Scholar (any year). A search on the term “electronic fruit” reveals nine documents. Most of these publications dealt with phantoms used for collecting data through sensors, e.g., pressure or acceleration, rather than mimicking features that are key for gripper testing (density, shape, firmness). For example, the design proposed by Defraeye et al. [6] also achieved similar size, shape, and colour of a real apple. However, their fruit did not mimic fruit density or firmness, which are important variables for gripper testing. In another application, artificial fruit were created to mimic the thermal response of real fruit (e.g., mango [7] and apple [6]). Some reports of model fruit are targeted to postharvest applications. For example, model fruit made of gels, e.g., 1% w/v gellan gel, have been used in studies on the rate of drying [8] (pp. 117–122) and rate of heating [9] (pp. 270–280). The Aweta acoustic firmness instrument is supplied with two silicone spheres of different densities for use in calibration of the unit [10]. The spheres mimic real fruit in density, weight, and firmness (as indexed by Young’s modulus), but not shape. The ImpacTrack™ is a model fruit equipped with accelerometers for the measurement of forces imposed on fruit during postharvest practices. The object is 3D-printed in two halves using a photopolymer resin to enclose the sensor module and mimic real fruit shape and weight, but not firmness or surface texture [11].

While there are many materials that could be used in production of a fruit phantom for harvesting trials, silicone (polydimethylsiloxane; PDMS) is a likely material for this application. PDMS is favoured for the creation of replicas of objects, given the fidelity of the cast (no shrinkage or expansion), the capture of fine surface details and the relatively benign casting process (e.g., no heat is generated). It has been used in the modelling of human organs for scientific studies. For example, Shanker et al. [12] reported on the use of PDMS to create a model of a human head for the study of the acoustic properties of the upper airway. In another example, Izdihar et al. [13] noted shrinkage problems with carrageenan-based polymer gels as compared to the use of PDMS in the fabrication of kidney phantoms for use in X-ray imaging.

PDMS has a density of 0.97 kg/m³ [14]. It is viscoelastic, i.e., it behaves as an elastic solid, similar to rubber, at short flow times. PDMS (also known as Silicone I) is a polymer of dimethyldichlorosilane (Me₂SiCl₂, syn. C₂H₆Cl₂Si) units, which are produced from SiO₂ and CH₂Cl₂ [15]. ‘Neutral cure’ silicone, also known as silicone II, is based on methyltrimethoxysilane (MeSi(OMe)₃). It is much slower to cure than silicone I and is therefore not recommended for mould creation [16].

The process of curing is based on cross-linking of the silane chains, increasing the molecular weights of the polymer and decreasing the viscosity to form a soft, compliant rubber [15]. The addition of a water donor, i.e., a material with many hydroxyl groups, such as starch or glycerine (as found in detergent), accelerates crosslinking, i.e., Si–O–Si networks, resulting in the ‘curing’ of the PDMS to a solid. Ceseracciu et al. [16] explains that the mechanical properties of the elastomer can be adjusted by variations in the proportions of starch and PDMS. Hao et al. [17] report on variations in storage modulus sensitivity (the ability of a material to store energy elastically) with a change in the mass ratio of starch and PDMS. Commercial preparations use proprietary formulations of PDMS, accelerants, fillers and other additives (e.g., patent US20010049427 [18]).

There are a large number of ‘popular press’ reports on the production of mouldable, putty-like material for craft projects using a formulation of readily available PDMS-based ‘silicone adhesives’ and starch [19]. Vegetable oil is used on mould surfaces or on the hands as a release agent, and oil-based paint is sometimes added to provide coloration.

1.3. Fruit Stalk Phantoms

The fruit is attached to a branch of the parent plant via a stalk (botanical term–peduncle). Harvesting involves breaking at either the fruit to stalk interface, or at the stalk to branch interface, depending on the fruit. There has been considerable work published on the measurement of, and modelling forces for, the detachment of fruit, particularly in the context of vibratory harvest systems, e.g., for citrus [20]. Bu et al. [21] developed a finite element analysis model of apple detachment to support robotic harvest work. A literature search on the keywords “model” and (“fruit stalk” or “stalk” or “peduncle”) did not retrieve any work on creating a model fruit stalk.

Empirical end-effector trials require both a phantom fruit and stalk, with a stalk attachment strength comparable to that of the actual fruit stalk. Assirelli et al. [22] report use of a ‘dynamometer’ (FA10, SAUTER, Balingen, Germany) to measure the force required to detach peach fruit from the branch. Polat et al. [23] used a dynamometer to measure the detachment force for pistachio nuts.

1.4. Aim

This report is intended as a technical note to assist researchers wishing to produce a low-cost fruit phantom from easily accessed materials, for use in harvest studies. A procedure was presented to produce a fruit cast, followed by the production of phantom fruit, with a comparison of materials for each step. A preliminary consideration of a phantom stalk was also presented. The phantom fruit characteristics were documented in terms of density and firmness, while the stalk was assessed in terms of detachment shear force. Mango fruit was chosen as the application example.

2. Materials and Methods

2.1. Mango Fruit

For the preparation of fruit casts, mango fruit of a range of sizes and shapes, covering the range encompassed by the main Australian cultivars of Kensington Pride, R2E2, Honey Gold, Calypso and Keitt were collected from farms during harvest periods and stored frozen at −20 °C. Except as mentioned, fruit were thawed prior to use, washed using tap water and dried with paper towel. For fruit stalk evaluation, fruit were collected at harvest maturity (i.e., dry matter content >15% of fresh weight, flesh colour of pale yellow)

2.2. Fruit Moulds

Two materials were trialed as casting agents for moulds–calcium sulfate hemihydrate (also known as gypsum plaster or plaster of Paris) (Uni-Pro, Kilsyth, VIC, Australia) and a PDMS-based product (Pinkysil™ Fast Set Silicone, Barnes, Moorebank, NSW, Australia). The two materials were compared in terms of cost, curing time, ease of use, durability and accuracy of the resulting object. Plaster of Paris is a traditional mould-making material because of its low shrinkage during drying. Pinkysil is described as PDMS with functional groups and auxiliaries for additional crosslinking, with a cured specific gravity of 1.1, Shore A hardness of 20 HA.

A fruit coated with Crisco Premium canola oil (Goodman Fielder Pty Ltd., Macquarie Park, NSW, Australia) as a release agent was placed in a container and covered with a slurry of calcium sulfate hemihydrate (100 g of plaster with 75 mL water) at ambient temperature. The container and fruit were agitated to remove air bubbles on the fruit. Three 2 mm diameter wires secured to an overhead frame were used to hold the fruit under the surface of the plaster by around 2 cm (Figure 1). The plaster was allowed to cure for one day, then removed from the container and cut equatorially to allow removal of the fruit. For use with liquid casting to create the phantom fruits, the mould halves were re-joined with a plaster mixture, and a 2 cm diameter hole drilled through the top of the mould. The small holes created by the wires were retained for air to escape during casting. Finally, the plaster mould was dried in an oven at 70 °C.

A Pinkysil^™ PDMS mould was created by introducing a 2 cm deep layer of the casting liquid into a cardboard container and allowing it to partly set before the introduction of a fruit and the addition of a further casting liquid to submerge the fruit. Wires were used to hold the fruit under the liquid, as for the plaster of Paris. The liquid was poured in a thin stream to avoid the introduction of air bubbles. As the PDMS set quickly, the mould was filled over several pours. The container and fruit were agitated to remove air bubbles on the fruit. The mould was left to cure for about four hours at ambient temperature when thawed fruit were used, or for 24 h in a cold room at 10 °C when frozen fruit were used. The mould was then cut equatorially to allow removal of the fruit (Figure 2).

2.3. Fruit Phantoms

Three materials were trialed for casting of the phantom fruit: (i) polyester resin (Clear Casting & Embedding Resin from Protite, c); (ii) Pinkysil™ Fast Set Silicone; and (iii) a mixture of a PDMS-based sealant (Wet Area Silicone, Selleys, Padstow, NSW, Australia) and corn or wheat starch (Black and Gold, IGA, Macquarie Park, NSW, Australia). Resin and Pinkysil™ were used in a plaster mould, whereas the PDMS sealant-starch mixture was used in both a plaster and a Pinkysil™ mould. PDMS liquid was not used directly in the Pinkysil™ mould due to adhesion between the cast and mould. The phantoms were assessed in terms of cost, curing time, ease of use, durability and size, shape, weight, and firmness of the cast relative to the original fruit.

Acrylic resin or Pinkysil™ liquid was poured into the plaster mould and allowed to cure at ambient temperature for 24 h. The mould was cut in half to remove the cured phantom fruit.

In the third approach, various ratios of silicone sealant–starch mixtures were trialed. A total mixture weight of 110% of the original fruit weight was prepared to accommodate losses during the mixing process and the higher density of this mixture in comparison to fruit. The materials were mixed using a spatula, then kneaded by hand until the mixture was uniform and its surface smooth. A small amount (<10 mL) of mineral oil (Johnsons Baby oil, Broadway, NSW, Australia) or detergent (Palmolive, Sydney, NSW, Australia) was added until the mixture no longer adhered to the spatula or hands. Preparation time varied between 15–30 min depending on the amount of material, as required, given the size of the phantom fruit. The mixture was then heaped into one half of the mould, covered with the second half of the mould and the two mould halves were secured with straps. Excess mixture overflowing the mould during closure was removed. The mixture was left to cure for about twenty-four hours at ambient temperature. The phantom was then removed from the mould, and any residual burr was eliminated (Figure 3).

The firmness of fruit near harvest maturity of the cultivars R2E2, Keitt, Kensington Pride and Honey Gold were assessed using a shore durometer with a type A tip (Gain Express, Hong Kong). An RMSE of 0.75 was obtained for measurements of a set of seven standards of known Shore value (range 29–88, SD = 22). Ten measurements were taken around the equator of each of five fruit per cultivar, involving three measurements on each of the flatter faces of the fruit, and two measurements on each of the narrow faces. There was no significant difference in the Shore value measurements between the sides of the fruit, so all ten measurements were averaged for each fruit.

The density of phantoms and real fruit was assessed from mass as measured using an Adam Equipment PGL 2002 scale and the volume was assessed from water displacement. To test brittleness, phantoms were dropped from a height of 1 m to a concrete floor and assessed for fracturing.

2.4. Fruit Peduncles (Stalk)

A preliminary investigation was undertaken involving two model stalks. The stalks were fabricated with the objective of matching the breakage shear stress to that of an actual mango stalk, as experienced when rotating the fruit to detach it from the stalk (Figure 4). The first design was based on a 2.5 mm diameter wooden dowel inserted at one end into the fruit phantom and at the other end into a tension spring (3/16″ × 1−3/4″) filled with silicone sealant and corn starch casting material. No gap was left between the spring and the fruit, so that on rotation of the fruit, the dowel breaking point was close to the fruit. The broken dowel was replaced after each rotation test. A variant on this design involved cutting a notch in a 3 cm diameter wooden dowel. However, it proved difficult to replicate the dimensions of the notch, and this technique was discontinued.

The second design employed a ribbon connected to a screw nut, and a pair of cylindrical rare earth (Neodymium) magnets. One magnet was fixed into the top of the model mango with silicone sealant, while the other magnet acted as a bridge to the nut. Magnets of different strengths were used to effect different stalk breakage forces (e.g., D-D8H3-N50 and 20014 from www.magnet.com.au, accessed on 2 November 2022). When rotating the fruit, connection failure between the magnets and the screw nut represented ‘stalk’ breakage.

A dynamometer (FT 327, Wagner Instruments, Greenwich, CT, USA) was used to measure the peak force required to detach the fruit from the stalk. The instrument’s hook was placed around the stalk, close to the fruit, and a force was applied normal to the fruit’s long axis to break the ‘stalk’. Measurements were made on real fruit and on fruit phantoms with either dowel or magnetic stalk designs.

3. Results and Discussion

3.1. Fuit Moulds

All materials used in creation of the mould were prone to entrapment of air bubbles during the creation of the mould. Gentle shaking or tapping of the mould container and use of a wooden skewer to dislodge bubbles resulted in the removal of most bubbles. If available, use of a vacuum chamber to degas the solutions is recommended.

Of the two materials trialed, plaster was the lowest-cost option and the easiest material to work with. However, several factors detract from its use: (i) the curing process is exothermic, causing damage to the fruit and impacting the accuracy of the phantom shape; (ii) the mould did not capture fine details of the object; (iii) the cured mould was relatively brittle and thus easily damaged; and (iv) a sealant coating was required to prevent the casting liquid from soaking into the plaster. Pinkysil™ PDMS was the most expensive and hardest to handle material, but it cured quickly, preserved fine surface detail, and had high durability (Figure 5). It was therefore chosen for use in creating the moulds.

A mould made of Pinkysil™ was three to four times heavier than the original fruit, with variations due to the volume of the cardboard container. A mould for a 500 g mango weighed around 1.7 kg and cost approximately AUD$103. The time required to create a mould varied depending on the temperature used to cure the mould: approximately 7 h if cured in a room at ambient temperature, or 27 h if cured in a cold room (

Table 1. Description of the approximate time required to produce a Pinkysil™ mould.

Process	Duration (h)
Creation of the cardboard container	2
Pinkysil™ addition to the container	0.75
Curing at room temperature	4
Curing at at 10 °C	24
Removal of mango from mould	0.5

).

3.2. Fruit Phantom

Of the three casting materials trialed, acrylic resin was eliminated based on its density (25% higher than actual fruit) and its brittleness, relative to fruit, while PinkySil™ PDMS was eliminated on the basis of its low firmness, relative to fruit, and the difficulty in working relative to the sealant PDMS–starch mixture (

Table 2. Fruit and phantom material characteristics. n is the number of fruits/phantoms used for the measurements. n/a means not applicable. Average and SD of n density measurements are presented. Firmness was assessed at 10 locations on each of n fruit, with average and SD presented. 10*n measurements.

Material	n	Density (g/cc)	Firmness (HA, Type A)	Brittleness	Surface Detail	Ease of Working
Fruit–Honey Gold (harvest stage)	3	0.94 (0.01)	73.7 (0.8)	n/a	n/a	n/a
–Kensington Pride (harvest stage)	4	0.93 (0.02)	73.1 (0.4)	n/a	n/a	n/a
–Keitt (harvest stage)	2	0.94 (0.01)	73.9 (0.4)	n/a	n/a	n/a
–R2E2 (harvest stage)	9	0.98 (0.01)	80.6 (1.2)	n/a	n/a	n/a
–R2E2 (eating stage)	3	0.95 (0.01)	43.9 (1.7)	n/a	n/a	n/a
Aweta reference ball 1	1	0.98	26.5	medium	n/a	n/a
Aweta reference ball 2	1	0.98	11	medium	n/a	a/a
Protite embedding acrylic resin	1	1.23	100	high	good	low
PinkySil PDMS	3	1.10	15.0 (0.5)	low	excellent	medium
Sealant + Corn Starch (3:2 ratio)	4	1.14 (0.01)	28.8 (1.3)	low	good	high
Sealant + Corn Starch (1:1 ratio)	1	1.14	31.0 (1.0)	low	good	high
Sealant + Wheat Starch (1:1 ratio)	1	1.14	33.2 (0.6)	low	good	high

). The resin was also the most difficult material to work with because of its viscosity and high emission of volatile compounds.

The reference fruit phantoms produced by the Aweta for their acoustic firmness equipment are intended to span the range of expected measurements in post-harvest sorting, i.e., from harvested fruit to ripening fruit. However, their firmer PDMS phantom returned a Shore value of only 36% of harvest-stage fruit, and 80% of the 1:1 silicone-starch phantoms (Table 2).

The silicone–starch mixture was adopted based on ease of working, low cost, ease of material availability, and fidelity of shape, density, firmness and surface texture to actual fruit (Figure 6). As the mango fruit is smooth, an image is provided of a citrus fruit phantom to demonstrate the fidelity of surface features (Figure 3a). However, these phantoms achieved less than half of the Shore values and were denser (by 16%) than at-harvest stage mangoes.

Various sealant-starch-detergent-oil mixtures were trialed (Table 1). Mixtures that included detergent resulted in soft and porous phantoms that were easily damaged or deformed, compared to the use of mineral oil. The addition of oil to the mixture resulted in slightly oily cast surfaces, which aided cast extraction from the mould while also reducing the difficulty to mix the components. Wheat starch produced phantoms of comparable firmness, surface quality and density as the phantoms based on corn starch. Increasing the proportion of starch in the admixture resulted in phantoms with higher firmness but with lower surface quality while also being harder to mix (Table 1). Mixtures with weight ratios of starch to silicone sealant approaching 1:1 were less sticky, such that small surface fissures could be present in the cast. These fissures were sealed using silicone sealant. Mixtures with more than 50% starch were difficult to mix and the resulting phantoms had a poor surface quality with multiple fissures. Mixtures with > 70% silicone were easier to produce but they were too soft (less than 25 HA). Phantoms made with the sealant–starch material were stable, with a mass loss of around 0.5 g over 100 days (Appendix A).

A mixture of 60–50% w/w silicone sealant and 40–50% w/w starch (of any source) is recommended to create a non-sticky, putty-like material that cures to a material of firmness comparable to pre-harvest fruit.

The cost to produce one 500 g phantom mango was approximately AUD$15 (250 g starch, 250 g silicone sealant, 10 mL of oil). The time required to create a phantom varied depending on its weight and its silicone–starch ratio (i.e., mixtures with higher concentration of silicone were quicker to make). It took between 15 to 30 min to create the mixture and put it in the mould, 24 h to cure the phantom, and another 20 min to release the phantom from the mould and remove burr.

3.3. Fruit Model Stalk

The measured average peak force to detach a mango (cv. Kensington Pride) fruit from its stalk was 6.5 kgf (using the unit used in post-harvest studies of fruit firmness). Similar detachment forces were achieved using either stalk phantom design, given the attention to the selection of dowel or magnet (

Table 3. Fruit detachment force of real and phantom fruit stalks (peduncles) based on dowel and magnets of different pull forces. Average and SD of n = 6 measurements are presented.

Material	Detachment Force (kgf)
Pre-harvest fruit	6.5 (1.0)
Phantom–wooden dowel	6.1 (0.6)
Phantom–magnet 1.5 kg	1.3 (0.1)
Phantom–magnet 10 kg	6.9 (1.4)

). However, since the PDMS–starch mixture has low elasticity, the repeated dowel replacement (as required in each use) resulted in damage to the fruit phantom. The magnet-based stalk phantoms were significantly easier to deploy, requiring less time to set up for each gripper test, and detachment force could be adjusted through choice of the magnet. The cost of the 10 kgf magnet was AUD$22.

4. Conclusions

A protocol was presented for the cost-effective production of fruit phantoms to support comparison trials of harvester end effectors. A phantom mango can be produced within 25 h for a cost of approximately AUD$15. A mould of fresh or frozen fruit can be produced using PinkySil™ PDMS or a similar product, with phantoms made by filling the mould with a 60:40 by weight mixture of acid-cure silicone sealant and wheat starch. The addition of a small amount of mineral oil or specialty release agent is recommended to improve workability and mould extractability.

The magnetic-latching stalk phantom appears promising, given the wide range of magnet strengths available, its low cost and ease of use. However, other stalk designs should be evaluated, e.g., an adhesive or a 3D-printed sacrificial member with an intentionally weak point of failure. Work on the orientation of magnet surfaces used in the phantom stalk is also recommended. Also, this preliminary study considered only peak shear force. Further studies should document the transient response involved in the stalk breakage of phantom compared to real fruit. Qualitatively, an increasing resistance is felt as real fruit rotate, until structural failure occurs.

Consideration of the coloring of the silicon–starch surface and procedures to embed pressure, accelerometer or other sensors in the phantoms is also warranted.