Real-Time Embedded Vision System for Online Monitoring and Sorting of Citrus Fruits

Abstract

Citrus fruits are the second most important crop worldwide. One of the most important tasks is sorting, which involves manually separating the fruit based on its degree of maturity, and in many cases, involves a task carried out manually by human operators. A machine vision-based citrus sorting system can replace labor work for the inspection of fruit sorting. This article proposes a vision system for citrus fruit sorting implemented on a dedicated and efficient Field Programmable Gate Array (FPGA) hardware architecture coupled with a mechanical sorting machine, where the FPGA performs fruit segmentation and color and size classification. We trained a decision tree (DT) using a balanced dataset of reference images to perform pixel classification. We evaluate the segmentation task using a pixel accuracy metric, defined as the ratio between correctly segmented pixels produced by a DT and the total pixels in the reference image segmented offline using Otsu’s thresholding algorithm. The balance between correctly classified images by color or size and their corresponding labels of that color and size evaluates the color and size classification algorithms. Considering these metrics, the system reaches an accuracy of 97% for fruit segmentation, 94% for color classification, and 90% for size classification, running at 60 fps.

1. Introduction

Nowadays, the food industry uses vision systems for fruit and vegetable classification. These systems commonly use conveyors to transport fruits through the sorting system [1]. The classifier system uses an interface to communicate with the actuators that perform the separation task. Generally, these systems determine the size, color, ripening, and quality of fruits [2,3,4,5,6,7,8,9,10,11,12].

Image processing techniques are used in agriculture to detect diseased leaf, stem, and fruit, to quantify the affected area by disease, to estimate or evaluate the productivity, among others, to find the shape of the affected area, to count or calculate the number of fruits entering in a sorting machine, to determine the color of the disease affected area, and to determine the size and shape of fruits. In this article, we propose a vision system for citrus fruits sorting by color and size, implemented on an FPGA device. The proposed hardware implementation on an FPGA of a classifier and vision tasks requires minimal resources, resulting in high performance. We coupled this implementation to an existing mechanical automatic citrus sorting machine.

We explored the hardware implementation of DTs classifiers, previously described in [13,14,15]. DTs require a minimum number of combinational and sequential components for their implementation. In parallel, we implemented an algorithm that processes a text-mode representation of a DT obtained from a dataset to generate the corresponding modules in Hardware Description Language (HDL) for later inclusion in a system that processes video in real time. We extracted the text mode representation of DTs using the Weka too, and we trained each DT using features obtained from images taken from citrus fruit videos captured in the cabin of the citrus separation system. The system allows the remote monitoring of the citrus fruit sorting machine’s performance.

From the implementations found in the state of the art, no system allows processing multiple lines with a low power consumption device. The most appropriate approach to achieve this goal is to find a classifier to train, among which DTs meet this objective, and a computing device with enough computational resources to achieve high-performance citrus fruit classification. It is also necessary for a classifier whose structure to represent the knowledge is relatively simple, discarding any classical or modern neural network from this objective.

The knowledge gap that this paper addresses is the dedicated digital hardware implementation of simple machine-learning techniques in complex problems, specifically in the issue of classifying citrus fruits by size and color, since deep-learning algorithms require vast amounts of images and classical algorithms are in disuse. We specifically chose a classifier based on DTs because their construction using digital components requires simple elements, and they represent a form of knowledge that is understandable by humans, facilitating debugging and extension.

We designed the citrus sorting system with low- and medium-sized industries in mind, where monetary resources are more constrained than the large industry. This work reports an implementation that adequately balances cost and performance. It is possible to achieve good accuracy using industrial PCs with GPUs, but power requirements and prices would increase. By using microcomputers, it is possible to lower the cost, but the classification accuracy and processing speed would be affected. The solution based on FPGAs manages, on the one hand, to have an intermediate cost between a workstation with GPU and a microcomputer. Existing fruit separation systems involve the processing of a single grading line. This work proposes a flexible solution that works simultaneously on multiple processing lines without compromising performance. Generally, one computer processes each grading line. To achieve this goal, we explored classifier simplification, which was possible by using a DT classifier trained with different datasets captured under the operating conditions of the citrus separator machine. The system guarantees real-time performance since the system processes each citrus grade line by a dedicated hardware block. Roughly speaking, FPGAs spend much less energy when compared with a conventional PC processor. Related to processing speed, even when the FPGAs work at lower frequencies than PC processors, the FPGAs use minimal hardware with low latencies and high throughputs.

The proposed system can handle multiple processing lines without compromising performance. The processing lines in the current implementation are limited to two video inputs available in the FPGA kit. The proposed approach can extend the processing capacity by replacing the FPGA with a higher-capacity device and adding multiple video inputs In a PC-based solution, the performance is low because of the workload on the processor and memory space for image storage. The designer must add one PC for each processing line to keep the same performance or use a more powerful PC.

This paper is structured as follows. In Section 2, related work is reviewed. In Section 3, the proposed system is described. In Section 4, the results are explained. Finally, in Section 5, conclusions are given and future work is addressed.

2. Related Work

In this section, we reviewed related works found in the literature. In Section 4.6, we compared the system proposed in this work to the most relevant reviewed systems.

Video streaming and image processing are useful tasks in classification systems. However, these tasks are computationally expensive in terms of time and computational resources. Video streaming provides information about the environment or gives useful visual features in visual quality inspection. Image processing techniques use these visual features as input to classification or clustering algorithms. Many IoT applications, such as surveillance video, healthcare, face recognition, human activities understanding, and farming, use video sensors [16,17]. Some approaches use low-cost and low-power machine vision systems [16]. However, most of these embedded video processing platforms exhibit low performance in real-time classification tasks due to their low computational power and bandwidth. In this case, GPU-based or FPGA-based approaches are suitable.

Machine vision-based fruit sorting systems are capable of replacing labor work for the inspection of fruit size. Seema et al. [18] reviewed fruit grading and classification systems. The authors summarize the most used features to identify the degree of rotting and ripening, the kind of fruits, and the machine-learning (ML) models used by the reviewed algorithms. They found two approaches: the first is multiple fruit identification systems focused on fruit differentiation, but the fruit quality is discarded. The training of these systems requires thousands of images of a series of different fruits. The second one, the specific fruit classification system, uses large image sets of a single fruit type to train and test the sorter. Although the first approach is more general, the second one is more suitable for single-type fruit sorting machines.

Concerning multiple fruit recognition approaches, we found several methods in the literature. Blasco et al. in 2003 [7] proposed a system to estimate the quality of oranges, peaches, and apples using four attributes: size, color, stem location, and detection of external blemishes. The proposed segmentation is based on Bayesian discriminant analysis, performing the correlation of fruit color using the colorimetric index values. The authors tested the classification system with apples, obtaining a blemish detection accuracy of 86% and size accuracy of 93%. Seng and Mirisaee [19] proposed an image retrieval method that combines classification models obtained from three features: color-based, shape-based, and size-based features to increase the accuracy of recognition. The proposed system uses the nearest neighbors classification to recognize 15 different fruits from their feature values, obtaining an accuracy of 90%. Jana et al. [12] proposed a system that preprocesses images to separate the fruit in the foreground from the background. Their system extracts texture features from the Gray-level Co-occurrence Matrix (GLCM) and statistical color features from the segmented image. The system creates a single feature descriptor from the extracted features and trains a Support Vector Machine (SVM) classification model. The generated model predicts the category for an unlabeled image from the validation set. The proposed method obtains an 83.33% overall accuracy. De Goma et al. [11] proposed a system to recognize fruits regardless using the K-nearest neighbor clustering based on statistical values of the color moments, GLCM features, and area by pixels for the size and shape roundness. They used a dataset with 15 different categories with 2633 images, obtaining an 81.94% accuracy.

Concerning orange fruit classification systems, we found several methods in the literature. Subramaniam and Balasubramanian [20] used parallel computing techniques on a multi-core processor to grade citrus fruits. They used the Task Parallel Library to add parallelism and concurrency to applications. They extracted geometrical features such as diameter, perimeter, area, and circularity under a laboratory-simulated real-time condition without a suitable conveyor. The system demonstrated the ability to estimate the diameter of the fruit with 98% accuracy. Sirisathitkul et al. [21] proposed an image processing technique to perform Chokun orange maturity sorting. In the training step, they captured images of 90 Chokun oranges of three different degrees of maturity with a color digital camera under normal illumination conditions. They performed an RGB to HSV color transformation for each image, using the hue colors to generate a set of decision rules. They tested the proposed model using 50 Chokun orange samples, obtaining a 98% accuracy. Chen et al. [22] proposed an orange sorting detection by obtaining four main features of the oranges, including fruit surface color, size, surface defect, and shape using image processing. They trained a BackPropagation neural network with these features. They report a sorting accuracy of 94.38%. Peter et al. [23] proposed an automatic system for disease identification in infected fruits images. The approach is evaluated on three diseases of the navel orange fruits, namely Citrus canker, Citrus melanose, and Citrus black spot, achieving 93% accuracy using global color histogram, local binary patterns, and Halarick texture features. Patel et al. (2019) [24] reported a system for orange sorting and detecting the bacteria spot defect based on four features: shape, size, color, and texture. They evaluated the SVM classification, obtaining a 67.74% overall accuracy. Behera et al. [25] proposed a system to grade oranges and identify deformities. They used a multi-class SVM with K-means clustering to classify orange diseases with an accuracy of 90%, and they used fuzzy logic to compute the degree of disease severity. Ifmalinda and Putri [26] proposed an orange sorting program based on diameter and skin color. They used diameter and RGB index to generate a set of rules to classify oranges, obtaining an overall accuracy of 87%. Wang et al. [27] proposed an algorithm to predict the sugar content of citrus fruits and performed a classification of the sugar content using light in the visible spectrum. Similar approaches for sorting apples can be found in [5,9,10,28]; for tomatoes in [23,29]; for sorting watermelons in [30]; for palm oil fruit sorting in [31]; and dates in [32].

Related to high-performance implementation using FPGA, there are few works. Martínez-Usó et al. in 2005 [8] proposed an unsupervised segmentation algorithm based on a multi-resolution applied to multi-spectral images of fruits as a quality assessment application. Lyu et al. [33] proposed a citrus flower recognition model based on YOLOv4-Tiny lightweight neural network using software and hardware co-design patterns. They generated the dynamic link library and integrated it into the FPGA-based embedded platform. The recognition accuracy of the citrus flower recognition model deployed on the embedded platform for flowers and buds was not less than 89.30%, and the frame rate was not lower than 16 FPS.

Zhenman et al. proposed an analytical model to compare FPGAs and GPUs performance. FPGAs can provide comparable performance or even achieve better performance than a GPU while consuming an average of 28% of the power required by a GPU for most Rodinia Kernels. Even when FPGAs use a lower clock frequency than GPUs, the FPGA usually achieves a higher number of operations per cycle in each computing pipeline due to its small pipeline initiation interval and considerable pipeline depth [34]. Zhang et al. proposed an FPGA acceleration of the generalized sparse matrix–matrix multiplication, an essential computing kernel for many algorithms in artificial intelligence [35]. They evaluated a Huffman tree scheduler on 20 real-world benchmarks, finding that the energy efficiency and performance are increased by 6× and 4×, respectively. Qasaimeh et al. assessed the energy efficiency of CPU, GPU, and FPGA implementation of computer vision kernels. They benchmarked algorithms for all the computer visions based on the OpenVX standard of GPU and FPGA platforms. Many simple seeds implemented on GPUs obtain a 1.1–3.2× energy/frame reduction. Still, the FPGA outperforms GPUs when complex ones that require a complete vision pipeline are necessary by obtaining a 1.2–22.3× energy/frame reduction [36]. Guo et al. performed a state-of-the-art review of neural network accelerator designs. They concluded that FPGAs achieve more than 10× better speed and energy efficiency than state-of-the-art GPU [37]. Sanaullah and Herbordt evaluated the hardware implementation of 3D Fast Fourier Transforms (FFTs) using OpenCL as Hardware Description Language. Their performance achieves an average speedup of 29× versus the current CPU and 4.1× versus the recent GPU [38]. Fowers et al. compared the performance and energy of sliding window applications when implemented on FPGAs, GPUs, and multicore devices. They concluded that FPGAS provides a significant performance increase in most cases, with speedups up to 11× and 57× compared with GPUs and multicores [39].

Recently, there have been efforts to use deep learning as an effective technique for fruit sorting. In [4], the authors propose a real-time visual inspection system for sorting fruits using a classification model obtained from state-of-the-art deep-learning convolutional networks. They test their system using apples and bananas. During real-time testing, the system obtained an accuracy of 96.7% for apples and 93.8% for bananas. For the training stage, they used a database composed of 8791 apples and 300 bananas of both healthy and defective fruits. Kukreja and Dhiman, in 2020 [40], proposed a dense CNN algorithm to detect the apparent defects of citrus fruit. They generated a first model without preprocessing and data augmentation on 150 images, achieving an accuracy of 67%. In a second model, the applied data augmentation and preprocessing after the model generation using 1200 images attained an accuracy of 89.1%. Sa et al. in 2016 [41] proposed an approach to fruit detection using deep convolutional neural networks, with application to automated harvesting using a robotic platform, completing fruit detection using imagery obtained from two modalities: color (RGB) and near-infrared (NIR). They compute both precision and recall performances, improving from 80.7% to 83.8% for the detection of sweet peppers. They created a model to detect seven fruits, which took four hours to annotate and train the new model per fruit. Leelavathy et al., in 2021 [42], proposed a CNN-based orange fruit image using a binary cross-entropy loss function, obtaining an overall accuracy of 78.57%. Hossain et al., in 2019 [43], proposed a framework based on two different deep learning architectures. The first is a proposed light model of six convolutional neural network layers, while the second is a fine-tuned visual geometry group-16 pre-trained deep learning model. They used two color-image datasets to evaluate their proposed framework. The first dataset contains clear fruit images, while the second dataset contains fruit images with noise, illumination, and pose variations, which are much harder to classify. Classification accuracies of 99.49% and 99.75% were achieved on dataset 1 for the first and second models, respectively. On dataset 2, the first and second models obtained accuracies of 85.43% and 96.75%, respectively.

Recently, existing solutions have used deep learning approaches to classify defects in fruits. In [43], the authors propose a system that classified orange images based on fresh and rotten using a CNN, with SoftMax classifier, using 800 orange images, achieving an accuracy of 78.57%. In [2], the authors generated a dataset of eight different classes of date fruits and compared several CNN models, such as AlexNet, VGG16, InceptionV3, ResNet, and MobileNetV2; MobileNetV2 architecture achieved an accuracy of 99%. In [44], the authors present a deep-learning system for multi-class fruit and vegetable categorization based on an improved YOLOv4 model that first recognizes the object type in an image before classifying it into one of two categories: fresh or rotten. Compared with the previous YOLO series, the proposed method obtained higher average precision than the original YOLOv4 and YOLOv3, with 50.4%, 49.3%, and 41.7%, respectively. In [45], the authors proposed an automatic image annotation to classify the ripeness of oil palm fruit and recognize a variety of fruits, trained with 100 images of oil fruit palm and 400 images of various fruits. From the previous systems, not many focus on classifying citrus fruits by color or size but focus specifically on fruit defects, which is a different problem than the one solved by the work reported in this paper.

3. Proposed System

We selected an FPGA for the proposed system in this work because they are suitable for implementing several parallelism degrees. Multiple parallelism degrees allow the system to achieve real-time performance, have low power requirements, the flexibility to test different architectures without changing the hardware, and the low latency required to interact with external actuators.

Related to the embedded implementation of image processing systems, FPGAs have proven to have higher performance compared to GPUs or embedded processors. Nakahara et al. implemented an FPGA’s YOLO (you only look once) object detector. They implemented a pipelined architecture that achieved 40.81 frames per second, which is 177.4× faster than an ARM Cortex-A57 and 27.5× faster than an Nvidia Pascall Embedded GPU [46].

3.1. Design Methodology

For the implementation of the FPGA-based citrus fruits monitoring and classification system, we propose the following design methodology:

• Citrus videos acquisition. In this phase, the operator must place a specific and varied quantity of fruit into the sorting machine to capture several videos of the fruit entering the processing chamber, preferably at different speeds.
• Assisted labeling of citrus images. In this phase, we developed an image-processing desktop application (IMDA) to perform the tasks required to process each image. The IMDA segments each image frame using the OTSU thresholding algorithm [47]. When the center of the segmented region is close to the center of the image, the IMDA stops and waits for the user’s response concerning the color and size classes. The IMDA asks the user for two labels for each image: its corresponding class related to maturity level (red or green) and its size class, which could be small, medium, or prominent in our system. Regarding the citrus fruit pixel database, for each image, the IMDA automatically assigns the label ‘1’ to the pixel that belongs to a citrus region and ‘0’ otherwise. To label the citrus fruits by size, we defined a metric based on the pixels covered by the citrus fruit in a circular region in the center of the image. The image processing requires field calibration to align the center of the camera’s field of view with the center of the conveyor rail. The purpose of the system was not to estimate the radius, diameter, or volume of the citrus fruit but to establish a rough reference to allow sorting of the citrus fruit by size. For this purpose, we used a circular object equivalent to a fruit weighing 300 g and established certain thresholds. A large citrus classification is assigned if the citrus fruit covers 100% to 75% of the predefined area. If the measurement falls, 75% to 50% is considered a medium citrus fruit, while for less than 50%, the system assigns the small citrus fruit category.
• Color space transformation and histogram computation. The IMDA performs the following color space conversions for each citrus image: RGB to HSV and RGB to Gray. The next step is to use the seven available channels (R, G, B, H, S, V, and Gray) to generate a 5-bin histogram, to be integrated into a 35-component vector for each citrus image. The IMDA adds the color class to the vector for its inclusion in the color dataset.
• Size generation. For each citrus image, the IMDA computes the area of each region of interest (ROI). The IMDA adds the size class based on the estimated ROI area for its inclusion into the size dataset.
• Automatic labeling of pixels for the citrus fruit. In this step, the IMDA generates a pixel dataset for each segmented image obtained in phase 2. The IMDA labels each pixel as a citrus pixel if it belongs to the ROI or non-citrus pixels in the opposite case.
• DTs training. With the pixel, color, and size data sets, the IMDA trains using a machine-learning tool to generate each DTs model. The knowledge extracted in each DT model is expressed in a text representation, using a human-readable format that is analyzed to create its corresponding hardware model.
• Base HDL model generation. All the required HDLs are created to be able to handle video signals to extract features and process the images.
• Hardware model for DT generation. The IMDA converts the DT model text representation to an implementable hardware description suitable to be integrated into the FPGA design.
• Testing and debugging in the field. Once the FPGA has the final configuration, we perform a test using the complete systems components and the generated models, where the model processes an image from a video signal, obtaining the classification by size and color.

3.2. Main Blocks of the Proposed System

Figure 1 shows the required hardware elements of the FPGA-based sorting system. Next, we give a description of each component and their interactions below:

• Microcontroller unit. This component acts as the controller device and performs all the machinery control, ignition, and recovery sequences, sending commands to actuators and reading from sensors, and reporting the sorting machine status and statistics to a cloud server through a wireless interface.
• FPGA Platform. This component hosts the core of the proposed vision system: the Citrus Fruit Classification Hardware Architecture (CFCHA). This component acts as a slave device, performing the tasks related to video processing and classification. The FPGA has a video port that captures and processes fruit images in real time, using an HDMI port for video transmission.
• PC. We use this component to analyze, develop, deploy, test, and configure the FPGA-based citrus sorting system. However, it is not part of the final system.
• Power driver block. This block sends commands to the actuators to move each citrus fruit to the appropriate container based on the class signal generated by the FPGA.
• Sensor block. This block detects citrus in the center of the processing chamber based on the data obtained by the sensor and reports to the MCU.
• Control panel. This panel contains elements that synchronize and activate the ignition sequences of the sorting machine.
• Cloud module. A web server hosts this module, which stores data sent from the microcontroller. This module also receives data requests from a client module to monitor the machine’s performance.
• Desktop client interface. A user executes this module to visualize sorting statistics about the citrus sorting machine.

To understand the functionality of each component, the processing flow is defined as follows:

• Citrus fruits enter into the processing chamber. The microcontroller continuously monitors the infrared proximity sensor and when a citrus is detected in the center of the chamber, a signal is sent to the FPGA.
• The FPGA processes the images of the current frame, generates the size and color classification results, and goes to stand-by state waiting to classify another fruit.
• The microcontroller receives from the FPGA the classification results and generates the proper signals to the actuators in order to move the processed citrus fruit to the appropriate container.

Figure 1 also shows the CFCHA main block (dotted rectangle). The proposed architecture uses two types of parallelism: spatial parallelism, where several blocks process data concurrently, and temporal parallelism, where a large task is divided into small phases, inserting registers between phases to store intermediate results and reduce the combinational delay path. The architecture includes combinational modules separated by a set of registers. The previous aims to reduce the combinational delay, avoiding synchronization problems and data degradation resulting from the frequencies at which the system works. Below we describe each CFCHA module:

• DVI Modules. These modules are the video interface from and to the FPGAs (DVI input and DVI output). The first generates the clock and synchronization signals, and the second integrates output video signals to creates the output video signals.
• Control Unit. The control unit synchronizes all the modules by sending and receiving control signals.
• Color conversion modules. These modules perform the required color space conversion (RGB to HSB and RGB to Gray).
• Pixel counter. This module receives the primary clock signal and the vertical and horizontal synchronization signals as input to keep the count of the currently processed pixel. It also determines the end of the processed frame and indicates the beginning of the next frame. Its output is the two-dimensional coordinates of the current pixel, which are helpful for other system modules.
• Histogram computation module. This module computes the histogram of the citrus to classify the citrus image by color.
• Color DT classifier module. This module classifies each citrus fruit based on the color histograms obtained in the previous processing phase using the DT previously obtained in the offline training.
• Size DT classifier module. This module estimates the area by counting the pixels that belong to citrus fruit. This area is next to the DT previously obtained in the offline training.
• Drawer module. This module is for the debugging phase. It replaces the corresponding pixel with a specific color to validate the correct functioning of the segmentation.

4. Results and Discussion

In this section, we will describe the hardware components, the main results of the hardware blocks of the proposed systems in terms of performance and resource utilization, the citrus fruit segmentation, the remote system monitoring, and a comparison with existing citrus grading systems.

4.1. Hardware Components

The main hardware components of the proposed system are described below:

• Microcontroller. This work uses an ATMEGA 2560 microcontroller with a Serial Peripheral Interface (SPI) for data exchange with the FPGA.
• Mechanical Citrus Fruit Grading Machine (MCFGM). This component transports the citrus fruits through an automatic circuit to where they will be processed. Figure 2a shows the main mechanical parts of the proposed system.
• Camera. The proposed system used a 1080p 60fps 2MP HDMI High-sensitivity Color HDMI AIBOULLY camera (model: HD-200CV) to capture citrus fruit live video to be processed by the FPGA. Figure 2b shows the camera inside the image acquisition chamber.
• FPGA development kit. We developed the proposed hardware architecture using a camera frame-buffer reference design of the AVNET Spartan-6 FPGA Industrial Video Processing Kit (IVPK). The core of the IVPK is a Xilinx Spartan-6 LX150T FPGA (model XC6SLX150T-3FGG676C). Adding the FMC-DVI and FMC-IMAGEOV Mezzanine boards provides the video interfaces necessary for developing video applications. Figure 2c shows the embedded processing platforms of the system.

4.2. Assisted Segmentation and Labeling of Citrus Fruit Images

Figure 3 shows different images of the semi-assisted segmentation process of citrus images that we used in the training stage of the segmentation DT. We captured ten videos using the camera mounted on the previously described chamber to extract the citrus images used to create the pixels dataset and histograms.

4.3. DT Hardware Implementation

To generate the digital hardware for the DT, we first train the dataset in Weka to obtain the text representation of the tree to be mapped to the hardware. Weka’s version of the DT is the one proposed by Quinlan [48]. In the second step, we coded a Python program that, from the text representation, generates the digital components equivalent to the generated DT. In the last step, the hardware architecture adds the HDL description of the tree as a component, which is finally analyzed and incorporated for the synthesis of the FPGA configuration file.

The hardware implementation of DTs is relatively simple, using a hardware description language. In Figure 4a, a simple DT exemplifies converting from graphic or textual representation to hardware blocks. Buffers and logic gates represent a delay unit (DU). Figure 4a shows a serial implementation of a DT based on the algorithms reported in [13,14,15,49,50], where we used comparator blocks with an enable input as a standard construction block. We fixed one of the inputs for each block (programmed at synthesis time), while the other inputs came from the other signals. The design requires two DUs for each stage and seven for the complete circuit. It is possible to eliminate a DU by replacing the first comparator with one without enabling it. However, this would generate non-homogeneous components, involving a risk for the data’s integrity or the programming phase. Based on state-of-the-art works [51,52,53,54], a fixed memory space for the hardware implementation for a DT is proposed in this work, using pipeline processing and an additional DU.

Figure 4c shows the design of the proposed parallel architecture that efficiently performs the classification using only three stages. The first one includes all the comparators without an enable pin, providing only one DU. The second one is the AND gates stage, which solves the DT branching. Finally, there is the OR gates stage, where all the possibilities where the output is class “1” are considered. This implementation has a maximum of four DUs, which eliminates three DUs compared to the serial performance shown in Figure 4b. If a deeper DT is required, the comparison stage would not add more DUs since we evaluated the comparison in the first parallel stage. In addition, we can reduce the branching of the second and last stages by using as many DUs as parallel connected AND gates, which would not be possible in a serial architecture.

The conveyor band machine used to capture the videos has three operating speeds: slow (3 m per minute), medium (6 m per minute), and fast (9 m per minute). To generate the initial dataset, we used a camera to capture three high-resolution videos at each possible speed of the conveyor band machine (for ten videos), each over three minutes in length. We considered the weight of 25 kg of various citrus fruits, which were recirculated several times through the conveyor belt to capture the videos. For each captured video, we analyzed the images in an OpenCV program on a PC, which, in an assisted way, obtained the frames based on the degree of maturation and the size of the fruit. The OpenCV program creates the pixels dataset and histograms from the citrus frames received for each video.

Table 1. Video features used to generate the datasets used in this work.

Video ID	Band Speed	Purpose	Length (mm:ss)
1	Low	Training	03:01
2	Low	Training	03:02
3	Low	Training	03:02
4	Middle	Training	03:02
5	Middle	Test	03:01
6	Middle	Test	03:01
7	Middle	Training	03:02
8	High	Test	03:02
9	High	Training	03:01
10	High	Training	03:18

shows the band’s speed, video length, and purpose. We used a random process to choose the videos for the dataset generation.

From the videos reported in Table 1, we obtained the image dataset for training and testing the DT. Each sample of this dataset could belong to two possible classes: class 1: citrus fruits of orange color, which implies that the fruit is very close to its optimum ripeness point, and; class 2: green citrus fruits, which means that the fruit is moving away from its optimum point of ripeness. The dataset used in tests 2 and 3 is subset of the set of videos used in test 1, randomly selected.

Table 2. Distribution of the citrus fruits color dataset and obtained training accuracies.

Dataset	Test 1		Test 2		Test 3
Training	Class 1	Class 2	Class 1	Class 2	Class 1	Class 2
	1151	210	578	104	346	63
	84.57%	15.42%	84.75%	15.25%	84.6%	15.4%
Total samples	1361		682		409
DT leaves	18		9		6
DT size	35		17		11
10-fold cross validation accuracy	95.73%		95.30%		94.62%
66–34% split accuracy	96.11%		93.10%		96.40%

shows that the obtained accuracies are similar for the performed tests.

Table 3. Distribution of the citrus fruits pixel dataset and the obtained training accuracies.

Dataset	Test 1		Test 2		Test 3
Training	Segmented	Non-segmented	Segmented	Non-segmented	Segmented	Non-segmented
	101,589	27,359	92,621	24,326	84,568	23,110
	78.78%	21.22%	79.2%	20.8%	78.53%	21.47%
Total samples	128,948		116,947		107,678
DT leaves	171		137		83
DT size	341		277		175
10-Fold Cross validation accuracy	99.27%		99.16%		99.29%
66–34% train-test data partition ratio	99.21%		99.14%		99.35%

shows the distribution of the dataset upon the values of the specific pixel, which determines if it belongs to citrus. In this case, two classes are: segmented (S) and non-segmented (NS). We created three test subsets from the assisted segmentation and generated a DT for each subgroup (test 1, test 2, and test 3, corresponding to the 100%, 90%, and 83% of the total dataset). This table also shows the obtained number of leaves, size, and accuracies, using 10-fold cross-validation and division of the set, using a 66–34% train-test data partition ratio, respectively.

Table 4. FPGA vs. PC accuracy comparison.

Test	Accuracy Type	PC Accuracy (%)	FPGA Accuracy (%)
Test 1	MIN	91.32	68.64
	AVRG	99.21	95.05
Test 2	MIN	87.21	87.08
	AVRG	97.5	97.52
Test 3	MIN	88.53	88.74
	AVRG	98.18	98.22

shows the accuracies (lowest and averaged) obtained for the segmentation model. The tests were performed over a dataset different from that used in the training stage, avoiding the possibility of obtaining an overfitted model. For test 1, the FPGA obtained the lowest accuracy, but in some cases, both platforms obtained an average accuracy above 95.05%. For test 2, the FPGA obtained the most insufficient accuracy, but in some cases, both platforms obtained an average accuracy above 95.5%. Finally, for test 3, the PC reached the lowest accuracy, but in some cases, both platforms got an average accuracy above 98.18%.

Table 2 shows the distribution of the training sets for the performed tests. The distribution is identical to the image dataset, with samples discarded due to eliminating duplicate samples. The precision obtained in each test reaches a value close to 95%. To test the generated models, we performed three tests:

• Test 1. A test was performed with the complete dataset.
• Test 2. A test was performed generating a model using only 50% of the total training set.
• Test 3. A test was performed generating a model using only 25% of the total training set.

Table 5. Accuracy comparison for the color sorting model implemented in both FPGA and PC.

	Test 1		Test 2		Test 3
	Class 1	Class 2	Class 1	Class 2	Class 1	Class 2
PC	96.42%	89.04%	94.1%	89.04%	97.05%	82.19%
	95.43%		93.43%		95.07%
FPGA	93.38%	91.89%	97.26%	85.14%	97.26%	75.68%
	93.44%		95.63%		94.35%

compares the accuracies of the computer-based and the FPGA-based implementations. Accuracies for both platforms are very similar, close to 95%, allowing us to infer that it could be possible to use fewer samples in the training stage of the DT to obtain an acceptable model without affecting the classification accuracy.

A software module generates images from the input videos mentioned in Table 1. For each dataset image, the system labels each citrus based on the citrus size and color. Later, the histogram is estimated for each image to generate the training and testing datasets.

Table 6. Size distribution of images used to generate the model for citrus sorting by size.

	Images for Each Class
Dataset	Small	Medium	Large
Train	388	941	61
	19.97%	48.45%	3.14%
	1390
	71.57%
Test	131	394	27
	6.74%	20.28%	1.4%
	522
	26.87%
Train + Test	519	1335	88
	26.72%	68.74%	4.53%
	1942

shows the distribution of each dataset class related to the citrus size.

Table 7. DT accuracy for each citrus size tested on the system.

Class	Small	Medium	Large
FPGA accuracy	86.26%	93.15%	77.78%
	90.76%

shows the accuracy obtained in the FPGA implementation using the proposed DT classifier, generated from a dataset for classifying citrus fruits by size. Concerning the reported accuracies, the large class obtained the lowest accuracy since it is the class that had the least number of samples, achieving an accuracy of 90%.

Table 8. Confusion matrix obtained for citrus size sorting.

	Small	Medium	Large
Small	118	13	0
Medium	27	367	0
Large	0	6	21

shows the confusion matrices for the citrus fruit classification by size. In the lower-left and the upper-right corners of each displayed matrix, it can be seen that the obtained model has no problems distinguishing between the Small–Large classes. On the other hand, the model has some difficulties in determining the Small–Medium classes, failing in a total of 40 cases of the tested dataset, which represents 7.24% of the samples. Nevertheless, among the Medium–Large classes, the performance is better, obtaining six failures, meaning 1.08% of the examples of the tested dataset.

Table 9. Hardware resource utilization of the complete architecture for citrus classification.

Resource	Used	Available	Use Percentage
Slices-Registers	764	184,304	0.41%
Slices LUT	31,222	92,152	33.9%

reports the required hardware resources to implement the proposed architecture in the FPGA. The obtained resources make it possible to process up to two processing lines independently with the same system, adding a second video capture board. Given the minimum number of reported records, we estimate that each architecture stage was used in the required pipeline structure, so their optimization is impossible. On the other hand, DTs use most of the reported LUTs.

Table 10. Accuracy variations for several confidence values and minimum node instances in the DT.

Confidence Factor	Minimal Instances	Accuracy	Leaf Number	Tree Size
0.25	2	99.27%	171	341
0.1	2	99.26%	108	215
0.25	10	99.18%	102	203
0.25	20	99.18%	30	119
0.25	40	99.13%	20	39
0.1	10	99.22%	69	137
0.1	20	99.17%	31	61

shows tests performed by modifying the confidence factor and the minimum number of instances per node, pruning parameters for the tree. As the confidence factor approaches zero and the minimum number of cases per node rises, the generated tree is smaller, but the accuracy is almost the same in all tests.

4.4. Accuracy Variations Due to the Conveyor Speed

Table 11. Classification accuracy of the proposed system for different camera resolutions and conveyor speeds.

Conveyor Speed	Camera Resolution	Classification Accuracy
Low	480p	91.15%
Medium	480p	89.75%
High	480p	87.21%
Low	720p	93.65%
Medium	720p	91.75%
High	720p	89.21%
Low	1080p	95.75%
Medium	1080p	93.65%
High	1080p	87.31%

shows the experimental results of tests performed at different image resolutions and conveyor band speeds. We estimated the classification accuracies for three conveyor speeds measured in Oranges Per Second (OPS). The OPS metric computes the average number of oranges that pass in front of the camera in a second, and only three speeds were selected, based on the current conveyor limitations, to evaluate the classification accuracy: 0.5 OPS (low), 2 OPS (medium) and 5 OPS (high). The results summarized in this table show that the resolution has a marginal impact on the classification accuracy and that a low resolution and a high OPS rate generate a lower precision. This lower accuracy is mainly due to the motion blur introduced at higher speeds, and a possible improvement is to add a motion-blur removal hardware block into the proposed hardware architecture. Another possible solution to this problem implies the addition of new hardware in the form of a stroboscopic illumination source synchronized with the image acquisition module to capture images without blur.

4.5. Remote Monitoring Citrus Fruits Classification

We use Python programming to implement this module and a PC with Python libraries installed and properly configured with an Internet connection to execute it. The execution of this script connects to the cloud module to obtain details of the monitoring status of the sorting machine. A text version of the monitoring report is available to obtain more detailed information about the sensor state of the citrus sorting machine and the number of citrus fruits classified in the day, including details of each fruit class.

4.6. Comparison of the Proposed System to Related Work

Table 12 shows the significant state-of-the-art works to which we compared the proposed approach. The discussion can be summarized as follows:

• Most of the related works use a PC as the main computing platform [21,22,23,24,25,43,55,56]. We proposed an FPGA due to the performance and power consumption advantages over conventional microcontrollers or even multi-core processors or GPU devices.
• Most of the related works use complex learning techniques, such as SVMs [22,24], Quadtrees [8], KNNs [56], and CNNs [43]. The Decision Rules are the most similar approach to ours, proposed in [21]. In our work, we used a DT classification technique, whose updating is fast and requires low computational resources for its hardware implementation.
• Few of the proposed works test their systems on a real sorting machine [26], and most of them focus on describing the machine-learning algorithms they use. In this work, we tested the proposed vision system on a simple sorting system, concentrating on one specific type of citrus (orange), but that left the possibility to work with multiple types of citrus.
• Most of the works in the literature use a combination of several technologies, such as Spectroscopes and NIR cameras [8,55,56], that, combined with RGB sensors, improve the accuracy, but make their systems much more complex and more expensive. On the other hand, we use a single RGB camera combined with a simple illumination system to assist the citrus segmentation and classification.
• Some works focus on detecting fruit defects rather than fruit sorting [25,42]. These methods assume that fruits are in the desired ripeness level, which could not be possible in a natural environment. We want to sort by size and color instead of finding fruit defects.
• Related to the citrus classification speed, in [26], the authors report a classification speed of 5.05 oranges per second, yielding up to 712 oranges per hour. Our work reaches the same classification speed but with better accuracy than the one reported in that work.

Table 12. Proposed system compared to existing systems in the state-of-the-art.

System	Computing Platform	Classifier Type	Fruit Classification Accuracy	Sensor Type	Remote Monitoring	Tested on a Grading Machine
[55]	PC	Voting	95%	NIR & RGB	NO	NO
[56]	PC	KNN	96%	Spectrograph & RGB	NO	NO
[8]	N/A	Quadtree	N/A	Spectrograph & RGB	NO	NO
[21]	PC	Decision Rules	98%	RGB	NO	NO
[22]	PC	SVM	95%	RGB	NO	NO
[24]	PC	SVM	67.74%	RGB	NO	NO
[26]	N/A	N/A	87%	RGB	NO	YES
[25]	PC	SVM	N/A	RGB	NO	NO
[23]	PC	RF, SVM and KNN	81%	RGB	NO	NO
This Work	FPGA	DT	94% (color), 90% (size)	RGB	YES	YES

Table 12. Proposed system compared to existing systems in the state-of-the-art.

System	Computing Platform	Classifier Type	Fruit Classification Accuracy	Sensor Type	Remote Monitoring	Tested on a Grading Machine
[55]	PC	Voting	95%	NIR & RGB	NO	NO
[56]	PC	KNN	96%	Spectrograph & RGB	NO	NO
[8]	N/A	Quadtree	N/A	Spectrograph & RGB	NO	NO
[21]	PC	Decision Rules	98%	RGB	NO	NO
[22]	PC	SVM	95%	RGB	NO	NO
[24]	PC	SVM	67.74%	RGB	NO	NO
[26]	N/A	N/A	87%	RGB	NO	YES
[25]	PC	SVM	N/A	RGB	NO	NO
[23]	PC	RF, SVM and KNN	81%	RGB	NO	NO
This Work	FPGA	DT	94% (color), 90% (size)	RGB	YES	YES

5. Conclusions and Future Work

This article reports a vision system for citrus fruits monitoring and sorting by color and size implemented on an FPGA device. This module receives input images taken by a camera placed in the processing chamber of a mechanical fruit sorting machine. The CFCHA performs the citrus fruit classification based on size and color and obtains an accuracy of 97% for the segmentation task, 94% for the color classification, and 90% for the size classification tasks. Information about the number of processed fruits can be remotely accessed using a PC.

The results showed that the proposed system, compared to those reviewed in Section 4.6, offers a good trade off between classification performance, cost, as well as power and hardware requirements.

There are several pending improvements to the proposed system. Using a larger FPGA and modifying the current citrus sorting machine would allow the system to classify multiple citrus lines with the same device in parallel, increasing the citrus sorting capacity of the overall design. Related to the classifier’s evaluation, since we used an unbalanced dataset to train the classifiers used in this work, the performance evaluation using metrics such as precision, recall, F1-score, and ROC curve would help us to propose a refined version of the current classifier.