Storage System for Automated Object Classification Based on Color Analysis ^†

Abstract

This article introduces a compelling study on the design and implementation of a storage system for automated object classification using color analysis. Despite notable advancements in robotic systems, the precise and reliable color-based classification in uncontrolled environments remains an ongoing challenge. The primary aim of this research is to develop an economical prototype that integrates color sensor and robotic arm to significantly enhance the efficiency of object-sorting processes across a wide range of industries. This study showcases the system’s impressive high accuracy rates of 97% for red, 96% for blue, and 98% for white objects in controlled settings. However, it also highlights the drop in accuracy to 76% for red, 64% for blue, and 68% for white objects in uncontrolled environments. These findings underscore the importance of maintaining consistent environmental conditions to optimize system performance, while also pointing to promising opportunities for modernization and semi-automation within the pharmaceuticals, food, and manufacturing sectors.

1. Introduction

Currently, object classification holds substantial importance across diverse fields, which require an autonomous process to be able to segment different elements of interest. In modern times, by utilizing techniques such as neural networks [1], machine learning algorithms like K-nearest neighbor [2], and image processing with computer vision [3], favorable results can be obtained to make an automatic classification machine, depending on the application you want to provide. Moreover, in robotic applications, automated color-based object classification systems can improve tasks like grasp form detection, contributing to the efficiency and reliability of robotic hand systems [4]. Overall, integrating automated object classification based on color analysis streamlines processes, saves time, and enhances accuracy in different domains.

Thus, taking into account that artificial intelligence is on the rise, offering powerful capabilities for color detection by taking advantage of machine learning models and convolutional neural networks to recognize subtle color changes imperceptible to the human eye [5,6,7], other simpler methods can be used, such as color sensors, as these can provide cost-effective color detection solutions by efficiently converting sensor signals to RGB colors [8]. While AI excels at complex color analysis and adaptability, a color sensor may be better suited for simpler applications where real-time processing speed and cost-effectiveness are prioritized [9].

Therefore, automated classification systems based on color analysis with sensors play a crucial role in various industries by enhancing efficiency, reducing labor costs, and improving accuracy. These systems enable the classification of products [10], such as objects [11], fruits [12], vegetables [13], and fish [14], based on their color attributes. By utilizing RGB sensors and color analysis techniques, these systems can accurately categorize items into different classes, detect defects, assess freshness, and sort products accordingly. The automation of this process not only increases productivity and quality assessment but also minimizes human error, speeds up decision-making processes, and facilitates remote monitoring and management through IoT connectivity. In general, automated classification systems based on color analysis with sensors are instrumental in streamlining operations, ensuring product quality, and optimizing resource utilization in various sectors [15].

2. Related Works

For this analysis, several references were explored, each adopting different approaches and methodological techniques to address research problems in the field of mechatronics. These studies employ methodologies such as experiments, controlled environment tests, and simulations, all aimed at solving specific problems.

In the context of developing classification stations, several notable concepts have emerged from the literature. The design and development of a delta robot for object classification presents a unique approach compared with conventional Cartesian robots. This application focuses on picking up and placing objects in stacking stations, and this paper demonstrates the accuracy achieved using artificial vision. However, it emphasizes that the shape of objects and the controlled environment are critical factors for success [16]. Robotic arms, irrespective of their type, showcase impressive capabilities in modern robotics. Their high degrees of flexibility enable them to maneuver in complex areas and assume challenging positions for effective object grasping. The versatility of robotic arms opens up various possibilities for industrial applications [17].

Implementing color sensors provides an affordable solution for identifying the color of objects during the classification process. Controlled by microcontrollers, these sensors enable the robotic arm to accurately determine colors using sophisticated analog readers. This low-cost development significantly enhances the efficiency and reliability of the object-picking process [18]. The integration of transportation bands or conveyor belts is essential for certain robotic arm manipulators. These bands facilitate the movement of objects to specific areas where the arm can easily grasp and relocate them to their designated stations. Integrating transportation bands with robotic arms optimizes the overall efficiency of the system [19]. The development of a human–machine interface (HMI) plays a crucial role in effectively controlling the programmable logic controller (PLC) and the entire robotic system. The HMI enables an interactive mode of operation within the station, allowing operators to monitor and control various parameters, ensuring seamless operation and precise control over the entire process [20].

In addition to the aforementioned ideas, other aspects have been considered in the literature. The use of a Cartesian arm, positioned as a square station on top of the working area, offers advantages in the classification process. It allows students to develop algorithms and control strategies more easily, enhancing their understanding and proficiency in working with robotic arms for effective object manipulation [21]. Furthermore, the accurate identification of object position is crucial in the classification process. The use of artificial vision to identify the color, shape, and position of the pieces has demonstrated 100% accuracy in all inspections conducted [22]. Open-source solutions, such as using a Raspberry Pi for color identification, have been explored. These solutions involve utilizing a camera to capture images, which are then analyzed pixel by pixel to identify RGB colors. Additionally, shape analysis is performed to determine the object’s position. This approach achieves accurate color readings and employs two microcontrollers, including an Arduino for robotic arm control [23].

3. Main Contributions

As the preceding section illustrates, storage systems are essential to automated color-based item classification. Nevertheless, several of these systems’ features have not been fully investigated yet. Thus, the following contributions are intended to fill in these gaps in this work:

• The machine uses color sensors to categorize objects based on their color, enhancing the efficiency of sorting processes.
• The system incorporates a cartesian arm that moves along the X and Z axes, combined with pneumatic actuators for precise object manipulation.
• This article explores a closed-loop control algorithm to regulate motor functions, ensuring accurate and reliable operation.
• The prototype integrates an HMI screen for monitoring and controlling the entire process, providing a user-friendly interface for operators.

This paper is structured as follows: Section 2 presents the methodology used to create the machine, detailing the components used for the mechanical, electronic, and computer parts, together with information about the operation of the machine. Furthermore, Section 3 shows the system in its final state and the results obtained in the experimental part with respect to the implementation of the color classification method, accompanied by the respective discussion. Finally, in Section 4, the conclusions obtained corresponding to the system implemented in this paper are presented.

4. Methodology

4.1. Mechatronic Design

Mechatronic design integrates mechanical, electrical, and information processing components to create innovative, functional, and streamlined systems [24,25,26]. Understanding this concept, the next section presents the mechatronic design used to develop the prototype, detailing its components and principal features for its correct operation.

4.1.1. Mechanical Design

This section describes the mechanism used to move the robotic arm in the three planes (X, Y, Z). First, structures are made in the upper part of the machine where the discs are placed for detection. Likewise, structures are incorporated to support the Cartesian arm mechanism in the same part of the machine (see Figure 1).

For the station, disc-shaped pieces were developed that were placed on the main upper structure to be classified. In this case, they were made up of two main components: The first is the lower disc that had a smaller internal diameter and the second component had a larger internal diameter (see Figure 2), which were joined under pressure. Its composition was made in this way for ease of manufacturing and for saving materials. The colors used for classification were white, blue, and red.

For the movement of the Cartesian arm, a base with wheels and a belt was used to move the arm horizontally from the left to the right. For movement along the Y axis, a pneumatic actuator operating at 0–10 PSI with a stroke of 50 mm was employed. This actuator, which uses compressed air, allowed the arm gripper to move forward and backward. Movement along the Z axis was achieved using a power worm mechanism with a corresponding nut, which was mounted on a support with wheels attached to the Cartesian arm. For holding objects, a parallel air gripper, operating at 14.5–101.5 PSI and with a diameter range of 20 mm (see Figure 3), was utilized for precise classification. This system ensures accurate and efficient movement and handling of the items.

4.1.2. Electronic Design

Referring to the electronic section of the machine, it is essential to highlight that an Xinje XC3-24RT-E PLC was employed to control various machine processes, powered by a 110 V electrical supply. Regarding the microcontrollers, two ESP32 units were utilized. The first microcontroller processes data from the TCS3472 color sensor via I2C communication for accurate color detection. Based on the detected hue, it sends signals to a corresponding relay module, facilitating the necessary voltage conversion from 5 V to 12 V, which is then transmitted to the PLC inputs. The second microcontroller is tasked with managing the DC motors in conjunction with an L298N H-bridge module. It activates the motors in response to the PLC outputs, which are also routed through relay modules to reduce voltage as needed.

Additionally, TCRT5000 proximity sensors are deployed to ascertain the position of the Cartesian arm base, while limit switches are installed on other axes to determine maximum positions along the X and Y axes. These signals are directed to the PLC inputs and, for controlling the flow of compressed air, two solenoid valves are activated by the controller outputs.

An HMI screen was integrated for process control, maintaining a constant interaction with the PLC. Moreover, power and emergency stop buttons, complete with their respective pilot lights, were installed. The electronic block diagram is depicted in Figure 4.

4.1.3. Software Design

Regarding the programming section, XCP-Pro V3.3 software was used for the PLC program and Arduino IDE V1.8.18 software for the microcontroller ESP32 using Ladder and C languages, respectively.

To perform the control and select the direction of the motors under the command of the ESP32, the process that can be observed in Algorithm 1 was established. It was used along with sequences based on timers from PLC programming, where each one had a certain time and, according to it each DC motor, was activated by the microcontroller. Therefore, first the Cartesian arm moves from the top to the bottom on the Z axis and then it moves from the left to the right on the X axis to place the piece in the structure of the respective color, thus ending the sequence. To move the motors and give them their respective direction, microcontroller pins connected to the H-bridge were established so that they rotated in one direction if one of the pins was “HIGH” and the other was “LOW”, and vice versa for the opposite direction.

Algorithm 1 Motor control.

Set variables while Motor is on do    if First, PLC timer equals 1 then      Move axis X rightwards    else if Second, PLC timer equals 1 then      Move axis X leftwards    else if Third PLC timer equals 1 then      Move axis Z upwards    else if Fourth PLC timer equals 1 then      Move axis Z downwards    end if end while

On the other hand, for the case of color-based detection and classification, the process is outlined in Algorithm 2. In this scenario, involving a maximum of three discs in the initial structure, variables were defined to determine the position of each disc. Consequently, if one of the three predefined colors was detected in the first position, the Cartesian arm would lower to that position and transfer the disc to the corresponding structure following the previously described algorithm. Upon completing this sequence, the process would repeat for each remaining object position.

Algorithm 2 Color detection ESP32.

Set variables while Sensor is on do    if Piece in position 1 then      Activate position variable 1      Take the piece      if Piece in position 1 is color blue then         Move the piece to a blue structure      else if Piece in position 1 is color white then         Move the piece to a white structure      else if Piece in position 1 is color red then         Move the piece to a red structure      end if    else if Piece in position 2 then      Activate position variable 2      Take the piece      if Piece in position 2 is color blue then         Move the piece to a blue structure      else if Piece in position 2 is color white then         Move the piece to a white structure      else if Piece in position 2 is color red then         Move the piece to a red structure      end if    else if Piece in position 3 then      Activate position variable 3      Take the piece      if Piece in position 3 is color blue then         Move the piece to a blue structure      else if Piece in position 3 is color white then         Move the piece to a white structure      else if Piece in position 3 is color red then         Move the piece to a red structure      end if    end if end while

Finally, a graphical HMI interface was developed within the same software to control the PLC, featuring three main functions: automatic, manual, and counting. In the case of automatic classification, there is an option to position the Cartesian arm at the starting point and initiate classification according to the previously explained programming. For manual classification, a panel with icons representing the directions the Cartesian arm can move, as well as object gripping actions, was developed. Lastly, the counting option allows visualization of the number of objects classified according to their color.

5. Results and Discussion

5.1. Machine Fabrication and Integration

The machine’s structure was developed and assembled internally, along with the cartesian arm system for sorting. The entire device can be seen in detail in Figure 5. As observed, the sorting structure is located at the top of the machine. Additionally, the control panel is located at the front of the structure, featuring various power and emergency stop buttons, as well as pilot lights to indicate if the machine is powered on (green) and if pieces are available (yellow). The HMI touch screen is located in the same section, allowing us to control the machine’s different processes, either manually or automatically. Primarily, the process for using the machine involves first turning it on, placing the pieces to be sorted at the starting point, and selecting the sorting mode. For a better understanding of the aforementioned process, the machine’s functionality can be visually appreciated at the following link, which shows the sorting process: https://youtu.be/IA7_wQ3IJ_M (accessed on 26 August 2024).

Regarding the mechanism and mobility of the Cartesian arm, no failures were observed in its movements across the three axes, achieving smoothness in the bases equipped with wheels and proper operation of the pneumatic actuator. Likewise, the electronic components ensure the correct functioning of the machine, including the motors and sensors used to determine the maximum and minimum positions of the mechanism. Additionally, the microprocessor and the PLC controller operated optimally to determine each of the sequences and processes for sorting the objects by their respective colors.

5.2. Objects Classification

For this section, tests of the sorting system were carried out in a controlled and uncontrolled environment, where 100 discs of each color were used for testing. Therefore, the lighting conditions were modified with a flashlight included in the gripper to lighten the colors of the parts. Thus, for the uncontrolled environment, different ambient light spectra were used to determine whether the system continued to function correctly. Regarding the graphics, the green bar indicates the iterations where the color was detected, while the red bar denotes the iterations where the color was not detected. Row 1 corresponds to the color red, row 2 to blue, and row 3 to white, as shown in Figure 6.

The color-based object-sorting machine demonstrated high accuracy in a controlled environment, with detection rates of 97% for red, 96% for blue, and 98% for white (see Figure 6a). However, in an uncontrolled environment, the accuracy significantly decreased, achieving only 76% for red, 64% for blue, and 68% for white (see Figure 6b).

The decreased system precision is due to uncontrolled lighting conditions during the storage system operation tests. To avoid this type of uncertainty, it is recommended to analyze the environment in which the storage operations will be executed based on the color of the objects. This would reduce problems in the classification process.

These results underscore the importance of maintaining consistent environmental conditions to ensure the system’s effectiveness.

6. Conclusions

In the research, it was possible to observe teams that had made similar prototypes. However, metrics still needed to be presented to validate the precision and accuracy of the system. In the present work, being able to use communication between different types of controllers has allowed for color classification, a human–machine interface, and their classification to be carried out, being an economic prototype applicable to the pharmaceutical, metalworking, mass consumption, and food sectors, among others. In these sectors, new technologies have been increasingly implemented in the form of modernization and semi-automation of initially manual processes, especially in SMEs. The color-based object-sorting machine demonstrated high accuracy in a controlled environment, with detection rates of 97% for red, 96% for blue, and 98% for white. However, in an uncontrolled environment, the accuracy significantly decreased, achieving only 76% for red, 64% for blue, and 68% for white.