A Review of the State-of-Art, Limitations, and Perspectives of Machine Vision for Grape Ripening Estimation ^†

Abstract

This work highlights the most recent machine vision methodologies and algorithms proposed for estimating the ripening stage of grapes. Destructive and non-destructive methods are overviewed for in-field and in-lab applications. Integration principles of innovative technologies and algorithms to agricultural agrobots, namely, Agrobots, are investigated. Critical aspects and limitations, in terms of hardware and software, are also discussed. This work is meant to be a complete guide of the state-of-the-art machine vision algorithms for grape ripening estimation, pointing out the advantages and barriers for the adaptation of machine vision towards robotic automation of the grape and wine industry.

1. Introduction

Grape harvesting is a great challenge that requires cost-effective solutions. The choice of harvest time determines the quality of the yield. Especially in wine production, deciding on the optimal harvest time concerning certain compounds, such as anthocyanins, is crucial in order to obtain optimal wine quality. Additionally, different vineyards ripen at different rates. Identifying maturity zones could enhance the efficiency of the harvesting operation. Grape harvesting based on grape ripening stage estimation increases the sustainable production of grape fruit products in the following three main directions: (1) improvement of the quality of harvested grapes due to homogenously ripened and equally fresh fruit, (2) reducing the post-harvest waste along the supply chain, and (3) reducing the production costs and human labor due to sustainable resource management.

In this context, an automated approach for grape ripeness estimation is considered necessary. Research focuses on the development of non-destructive, cost-effective, and environmentally friendly techniques. Machine vision has recently gained considerable acceptance for agricultural-related tasks. To this end, machine vision has been employed in in-field practical applications for the assessment of grape ripeness. The reported results reveal that image processing has good potential to be used for grape ripeness estimation, not as a substitute, but rather as an attractive alternative to chemical analyses, due to its simplicity, flexibility, and low cost. The aim of this work is to increase the insight into the application of machine vision techniques for grape ripening estimation. The techniques reported here are recent, and could be utilized as effective and rapid tools for decision making regarding the desirable harvest time.

2. Machine Vision Methods for Grape Ripeness Estimation

Different grape cultivars display a different refractometric index, which is related to maturity. For instance, table grapes must have at least 16° Brix to be considered ripened, while Sauvignon Blanc may ripen at 20–22° Brix, etc. However, even this is relevant, since the proper harvesting time is not necessarily related to a standard ripening value, but to a desired ripening value depending on the harvested variety. In the wine industry, the health condition and maturity of harvested grapes need to be evaluated in order to determine the exact procedure, diffusional, enzymatic, or biochemical, to be applied. Objective criteria to judge the ripeness stage of grapes are those related to chemical attributes, such as titratable acidity (TA), soluble solid content (SSC), etc. Subjective maturation criteria are just as important, and include the sensory characteristics chosen to be discriminative among samples.

During ripening, many physical and biochemical changes occur that affect grape characteristics. Usually, red–green–blue (RGB) imaging is the most cost-effective way to determine color channel values and other characteristics, such as shape and texture. The RGB color space depends on the device on which it is displayed; therefore, it is not adequate when it comes to absolute measures. For this reason, the CIELAB color space is used as an international standard for color measurements. However, alternative color spaces have also been investigated, such as hue–saturation–intensity (HIS), hue-saturation value (HSV), etc. Classic near-infrared (NIR) spectroscopy has been proven as a powerful analytical tool to designate bioactive compounds in grapes. The main advantage of NIR spectroscopy over other methods is the chemical-free sample preparation and the ability to determine efficiently optical properties of the fruit that are strongly related to the chemical and physical properties, and, thus, to maturity. Hyperspectral imaging is also considered to be an evolving tool, since it combines conventional imaging with spectroscopy, and, thus, one can obtain both the spectral and spatial information of an object. Table 1 summarizes the recent applications of hyperspectral imaging, color imaging, and NIR spectroscopy for grape ripeness estimation.

Table 1. Applications of hyperspectral imaging for grape ripeness estimation.

Cultivar	Maturity Index	Spectral Range/Color Space	Number of Images	Prediction Model	Evaluation	Ref.
Touriga Franca (TF), Touriga Nacional (TN), Tinta Barroca (TB)	pH, anthocyanin	380–1028 nm	225	Neural Network (NN)	pH R2: 0.723 (TF), 0.661 (TN), 0.710 (TB) Anthocyanin R2: 0.906 (TF), 0.751 (TN), 0.697 (TB)	[1]
cv. Malagousia	TSS, pH	510–790 nm	12	Multiple linear regressor (MLR), support vector machine (SVM)	Classification rates: TSS 83.33% (MLR), TSS 83.33% (SVM), pH 75% (MLR), pH 75% (SVM)	[2]
Syrah, Tempranillo	Flavanols, total phenols	900–1700 nm	200	principal component analysis, modified partial least squares, K-means, linear discriminant	Classification rates: 83.3% (leave-one-out cross-validation), 76.9% (external validation)	[3]
Vitis vinifera L. cv., Tempranillo, Syrah	Anthocyanins	900–1700 nm	99	Principal component analysis	0.86 R2	[4]
Kyoto grapes	SSC, pH	HIS, NTSC, YCbCr, HSV, CMY	180	Multiple linear regressor (MLR), partial least-squares regressor (PLSR)	Mean square error: pH 0.0987 (MLR), pH 0.1257 (PLSR), SSC 0.7982 (MLR), SSC 0.9252 (PLSR)	[5]
<Not defined>	Visual assessment	RGB, HSV	289	Dirichlet mixture model (DMM)	125.04 perplexity, 0.29 average perplexity per color	[6]
Italia, Victoria	Visual assessment	HSV, CIELAB	800	Random forest (RF)	Cross-validation classification accuracy: up to 100% (Italia), up to 0.92 (Victoria)	[7]
White grape Sonaka	Visual assessment	RGB HSV	4000	Convolutional NN (CNN), support vector machine (SVM)	Classification rates: 79% (CNN), 69% (SVM)	[8]
Cabernet Sauvignon	Visual assessment	RGB	13	Neural network (NN)	Average error: 5.36%	[9]
Syrah, Cabernet Sauvignon	TSS, flavonoid, anthocyanin	RGB	1008	Convolutional neural networks	Classification rates: 93.41% (Syrah), 72.66% (Cabernet)	[10]

A limitation of the use of machine vision algorithms in grape-related applications is attributed to the lack of large-scale public datasets of grape bunch images for testing innovative methodologies and performing comparative evaluation reports. Researchers must rely on the data collected by themselves, which are neither universal nor comparable. The complexity of agricultural environments in terms of diversity and dynamically changing conditions, such as illumination and vegetation, are the main obstacles that machine vision technology attempts to overcome. Robust algorithms, able to operate in heterogeneous environments, need to be developed. The application of visual technology in the field involves the integration of multiple parameters and principles that can only be encountered in natural setups, in practical applications, and not in laboratory settings. In recent years, the development of graphical peripheral units (GPUs) helped to increase the computational power and allowed powerful real-time pattern recognition techniques to be used on-site. In the future, machine vision algorithms are expected to play a vital role in sustainable agricultural automation, towards improving local economies and promoting ecology.

3. Conclusions

Nowadays, fruits are still harvested manually by fruit pickers, based on subjective criteria and extensive sampling, followed by chemical analyses. Maturity is critical for the storage life of harvested fruits, and the quality of table grapes and produced wines. The degree of maturity defines the way grapes are further processed. Machine vision algorithms are employed for grape ripeness estimation, as a non-destructive, labor-saving, cost-effective, and eco-friendly alternative. This work explores the possible applications of machine vision technology in grape ripeness estimation, and discusses the limitations and challenges hindering their successful adoption for sustainable agricultural automation.