*3.2. Object Recognition*

Object recognition is a typical issue in industrial robotics applications, such as sorting, packaging, grouping, pick and place, and assembling (Table 3). The appropriate recognition method and equipment selection mainly depends on the given task, object type, and the number of recognisable parameters. If there are a small number of parameters, simpler sensing technologies based on typical approaches (geometry measuring, weighing, material properties' evaluation) can be implemented. Alternatively, if there are a significant number of recognisable parameters, photo or video analysis is preferred. Required information in two- or three-dimensional form from image or video can be extracted using computer vision techniques such as object localisation and recognition. Various techniques of visionbased object recognition have been developed, such as appearance-, model-, template-, and region-based approaches. Most vision recognition methods are based on deep learning [34] and other machine learning methods.

In a previous study [35], a lightweight Franka Emika Panda, cobot with seven degrees of freedom and a Realsense D435 RGB-D camera, mounted on an end effector, was used to extend the default robots' function. Instead of using a large dataset-based machine learning technique, the authors proposed a method to program the robot from a single demonstration. This robotic system can detect various objects, regardless of their position and orientation, achieving an average success rate of more than 90% in less than 5 min of training time, using an Ubuntu 16.04 server running on an Intel(R) Core(TM) i5-2400 CPU (3.10 GHz) and an NVIDIA Titan X GPU.

Another approach for grasping randomly placed objects was presented in [36]. The authors proposed a set of performance metrics and compared four robotic systems for bin picking, and took first place in the Amazon Robotics Challenge 2017. The survey results show that the most promising solutions for such a task are RGB-D sensors and CNN-based algorithms for object recognition, and a combination of suction-based and typical two-finger grippers for grasping different objects (vacuum grippers for a stiff object with large and smooth surface areas, and two-finger grippers for air-permanent items).

Similar localisation and sorting tasks appear in the food and automotive industries, and in almost every production unit. In [37], an experimental method was proposed using a pneumatic robot arm for separation of objects from a set according to their colour. If the colour of the workpiece is recognisable, it is selected with the help of a robotic arm. If the workpiece colour does not meet the requirements, it is rejected. The described sorting system works according to an image processing algorithm in MATLAB software. More advanced object recognition methods based on simultaneous colour and height detection are presented in [38]. A robotic arm with six degrees of freedom (DoF) and a camera with computer vision software ensure a sorting efficiency of about 99%.

A Five DoF robot arm, "OWI Robotic Arm Edge", proposed by Pengchang Chen et al., was used to validate the practicality and feasibility of a faster region-based convolutional neural network (faster R-CNN) model using a dataset containing images of symmetric objects [39]. Objects were divided into classes based on colour, and defective and nondefective objects.

Despite significant progress in existing technologies, randomly placed unpredictable objects remain a challenge in robotics. The success of a sorting task often depends on the accuracy with which recognisable parameters can be defined. Yan Yu et al. [40] proposed an RGB-D-based method for solid waste object detection. The waste sorting system consists of a server, vision sensors, industrial robots, and rotational speedometer. Experiments performed on solid waste image analysis resulted in a mean average precision value of 49.1%.

Furthermore, Wen Xiao et al. designed an automatic sorting robot that uses height maps and near-infrared (NIR) hyperspectral images to locate the region of interest (ROI) of objects, and to perform online statistic pixel-based classification in contours [41]. This automatic sorting robot can automatically sort construction and demolition waste ranging in size from 0.05 to 0.5 m. The online recognition accuracy of the developed sorting system reaches almost 100% and ensures operation speed up to 2028 picks/h.

Another challenging issue in object recognition and manipulation is objects having an undefined shaped and contaminated by dust or smaller particles, such as minerals or coal. Quite often, such a task requires not only recognising the object but also determining the position of the centre of mass of the object. Man Li et al. [42] proposed an image processing-based coal and gangue sorting method. Particle analysis of coal and gangue samples is performed using morphological corrosion and expansion methods to obtain a complete, clean target sample. The object's mass centre is obtained using the centre of the mass method, consisting of particle removal and filling, image binarization, and separation of overlapping samples, reconstruction, and particle analysis. The presented method achieved identification accuracy of coal and gangue samples of 88.3% and 90.0%, and the average object mass centre coordinate errors in the x and y directions were 2.73% and 2.72%, respectively [42].

Intelligent autonomous robots for picking different kinds of objects were studied as a possible means to overcome the current limitations of existing robotic solutions for picking objects in cluttered environments [43]. This autonomous robot, which can also be used for commercial purposes, has an integrated two-finger gripper and a soft robot end effector to grab objects of various shapes. A special algorithm solves 3D perception problems caused by messy environments and selects the right grabbing point. When using lines, the time required depends significantly on the configuration of the objects, and ranges from 0.02 s when the objects have almost the same depth, to 0.06 s in the worst case when the depth of the tactile objects is greater than the lowest depth but not perceived [43].

In robotics, the task of object recognition often includes not only recognition and the determinaton of coordinates, but it also plays an essential role in the creation of a robot control program. Based on the ABB IRB 140 robot and a digital camera, a lowcost shapes identification system was developed and implemented, which is particularly important due to the high variability of welded products [44]. The authors developed an algorithm that recognises the required toolpath from a taken image. The algorithm defines a path as a complex polynomial. It later approximates it by simpler shapes with a lower

number of coordinates (line, arc, spline) to realise the tool movement using standard robot programming language features.

Moreover, object recognition can be used for robot machine learning to analyse humans' behaviour. Such an approach was presented by Hiroaki et al. [45], where the authors studied the behaviour of a human crowd, and formulated a new forecasting task, called crowd density forecasting, using a fixed surveillance camera. The main goal of this experiment was to predict how the density of the crowd would change in unseen future frames. To address this issue, patch-based density forecasting networks (PDFNs) were developed. PDFNs project a variety of complex dynamics of crowd density throughout the scene, based on a set of spatially or spatially overlapping patches, thus adapting the receptive fields of fully convolutional networks. Such a solution could be used to train robotic swarms because they behave similarly to humans in crowded areas.

**Table 3.** Research focused on object recognition in robotics.



**Table 3.** *Cont.*

A few main trends can be highlighted from the research analysis related to object recognition in robotics. These can be defined as object recognition for localisation and further manipulation; object recognition for shape evaluation and automatic generation of the robot program code for the corresponding robot movement; and object recognition for behaviour analysis to use as initial data for machine learning algorithms. A large number of reliable solutions have been tested in the industrial environment for the first trend, in contrast to the second and third cases, which are currently being developed.
