Application of Vision Algorithms to the Problem of Recognizing the Level of Blood in an Aliquoting Tube Biosample ^†

Abstract

This work is devoted to the analysis of the application of object detection algorithms for recognizing the level of blood in a test tube. Based on the initial data and data received from the sensory system, pattern recognition is performed and decisions are made to complete the task. It is necessary to control the level of the liquid being determined in the test tube to solve the technological problems of biomaterial aliquoting. In this case, it should be taken into account that the liquid has two fractions that differ in color: a blood clot and serum. The glare of the tube surface and lighting features should also be taken into account. These factors greatly complicate the operation of the vision system. Software in the Python language has been developed that implements the ability of a vision system to recognize the level of blood in test tubes and visualize the recognized blood level in several ways: contour selection, dot marking, and color filtering. The developed methods are supposed to be applied for biosample aliquoting using a delta robot in a multirobotic system, which will increase the productivity of ongoing biomedical research through the application of new technical solutions and principles of intelligent robotics. The visualized results of the work of the programs under consideration are presented, and an assessment of the quality of recognition implementation is given.

1. Introduction

Vision systems are applied to automate many production processes, including those for biomaterial aliquoting, where it is necessary to control the level of the liquid to be determined in the container. To solve this problem, the use of vision algorithms has been proposed to determine the fluid contour with subsequent interpretation of the information received.

Research has been ongoing for quite some time in the field of border and contour recognition in an image. Technical vision can be used in various fields; for example, a study iwa made of the possibility of using technical vision for segmenting abnormal skin layers in computer analysis of images in [1]. In this work, the process of extracting asymmetric patterns from dermoscopic images is separated using HSV segmentation to find the contour image. Automatic RGB-HSV separation is used, which segments the skin lesion. The proposed automatic segmentation can be useful for a dermatologist in identifying affected areas.

The article [2] discussed the possibility of detecting human skin using the RGB (red, green, blue), HSV (hue, saturation, value) and YCbCr (brightness, chroma) color models. This article proposes a new algorithm for detecting human skin. The objective of the proposed algorithm is to improve the recognition of skin pixels in given images. The algorithm takes into account not only individual ranges of three color parameters, but also combination ranges, which provides greater accuracy in recognizing a skin area in a given image. Two automatic methods for detecting bleeding in videos of wireless capsule endoscopy of the small intestine using different color spaces were proposed in the article [3]. The first method works pixel-wise and is based solely on the color. The second method uses a more sophisticated approach that not only relies on pixel colors but also assumes that the blood in the frame forms a continuous area (or several such areas), which gives an idea of the shape and size of the blood spot.

Additionally, article [4] proposes a detection algorithm that takes advantage of the camera and downloaded data to determine color based on RGB values. The algorithm involved calls on a function that runs loops on readjusting the distance based on a nearest match. This effortlessly helps define a color based on the RGB color space with a peaking accuracy.

Articles [5,6] proposed color estimation methods based on the HSV model. These methods will translate the RGB values of the points in video images to HSV values and use HSV values to recognize the color. The developed software for real-time video object recognition based on color features is presented, which has achieved the goal of real-time video motion detection and object color recognition. It can be said that the algorithms are accurate and similar to human motion detection when watching a video, which demonstrates the good performance of target identification and color estimation by the program.

Various color models and methods are used to detect colored objects using technical vision, and they have both advantages and disadvantages. Within the framework of this article, an algorithm is proposed based on the HSV color model for blood recognition in the process of performing the technological process of aliquoting biosamples using the proposed multirobotic system.

2. Synthesis of the Algorithm

Robotic biomaterial aliquoting [7] involves a Uni robot manipulator based on a serial robot with six rotational joints and a gripper that moves test tubes to an area where a parallel DeLi robot based on a 4-DOF delta robot with a picking head performs the aliquoting process (Figure 1).

The blood in the tubes is divided into two fractions, and DeLi has to collect only one of the fractions. The level of each of the fractions may be different for each of the test tubes. The use of technical vision in a multirobotic system for biomaterial aliquoting will eliminate the routine and labor-intensive operation of separating a blood clot from serum by an operator at the sample preparation stage, which will reduce the use of consumables, the risk of errors, and contamination of the surfaces of laboratory equipment. In this regard, the problem arises of recognizing the contour and level of the required blood fraction. To solve this problem, an algorithm based on the HSV color model is proposed.

The color model is cylindrical but can be represented as intervals of H, S and V values ranging from 0 to 1,0. The synthesized algorithm is based on using I recognition training data to form an A array describing the distribution of HSV values on the training data. The first part of the algorithm consists of forming the array A. The set of all possible combinations of HSV values are grouped into $N^3$ blocks. During cycles over the height and width of the image, the values $H,S$ and V are determined for each of the pixels. Next, the value of the cell $i,j,k$ of the A array, corresponding to the block containing the value of $H,S$ and V for this pixel, is incremented. Pixels with values $H_B,S_B$ and $V_B$ corresponding to the background of the training image are not taken into account. In the second part of the algorithm, cycles are performed over the image D, on which it is required to determine the fluid boundary. If the value $H,S$ and V of a pixel corresponds to a cell $i,j,k$ of the array A, which has a sufficient value r of repeatability, then the counter c is incremented, showing the number of suitable pixels in a horizontal line. If the counter value reaches the value of p, then this horizontal line is the boundary of the fluid.

The synthesized Algorithm 1 works as follows:

Algorithm 1: Liquid Level Detection Using Training Dataset.

Input:$I,D,N,r,p,H_B,S_B,V_B$ 1: $A_{i,j,k}=0,i\in [0;N-1],j\in [0;N-1],k\in [0;N-1]$ 2: for$x=0,\dots ,{\mathtt{Height}}_I$do 3:     for $y=0,\dots ,{\mathtt{Width}}_I$ do 4:         $H=I_{x,y}^{\left(0\right)},S=I_{x,y}^{\left(1\right)},V=I_{x,y}^{\left(2\right)}$ 5:         if $H\ne H_B$and$S\ne S_B$and$V\ne V_B$ then 6:            $i=\lfloor H·N\rfloor ,j=\lfloor S·N\rfloor ,k=\lfloor V·N\rfloor$ 7:            $A_{i,j,k}=A_{i,j,k}+1$ 8:         end if 9:     end for 10: end for 11: $y=0$ 12: while$y\le {\mathtt{Width}}_D$and$\mathtt{Finish}=$false do 13:     $c=0$, $x=0$ 14:     while $x\le {\mathtt{Height}}_D$and$\mathtt{Finish}=$false do 15:         $H=D_{x,y}^{\left(0\right)},S=D_{x,y}^{\left(1\right)},V=D_{x,y}^{\left(2\right)}$ 16:         $i=\lfloor H·N\rfloor ,j=\lfloor S·N\rfloor ,k=\lfloor V·N\rfloor$ 17:         if $A_{i,j,k}>r$ then 18:            $c=c+1$ 19:            if $c>p$ then 20:                $y_l=y$, $\mathtt{Finish}=$true 21:            end if 22:         end if 23:         $x=x+1$ 24:     end while 25:     $y=y+1$ 26: end while

Depending on the amount of training data and the proximity of the HSV values of the environment to the HSV values of the recognized object, the value of the required r repeatability level of the A array block in the I training data is selected. The number of pixels p in the recognizable image D corresponding to the upper border of the blood in the tube should be selected based on the number of horizontal pixels corresponding to the width of the tube in the image.

3. Simulation Results

The synthesized algorithm was implemented in the Python programming language. Determination of HSV values for image pixels and rendering was performed using the OpenCV library. The algorithm was implemented for two scenarios. As part of the first scenario, photographs of test tubes with blood were used for training and recognition. In the second scenario, training and recognition were performed on the data received from the webcam. Another feature of the second scenario is the use of a liquid imitating blood.

3.1. First Scenario

In the first scenario, an image with fragments of photographs of real blood in test tubes was used as training data, as shown in Figure 2a. The fragments were placed on a black background. In this case, $H_B=S_B=V_B=0$.

The following initial data were used for blood recognition: $N=32,r=40,p=35$. The recognition results are shown in Figure 3. As can be seen from the figure, the algorithm made it possible to identify both the blood level in the test tube and its contour.

3.2. Second Scenario

To obtain fragments used as training data, as shown in Figure 2b, as well as images for recognition, a webcam with a resolution of 1920 × 1080 pixels was used.

A test tube with a liquid simulating blood was located at a distance of about 185 mm from the webcam during recognition; the test tube diameter was 15 mm. In this case, the width of the test tube in the image received from the webcam was 119 pixels. The following initial data were used for blood recognition: $N=32,r=500,p=35$. The recognition results are shown in Figure 4.

As can be seen from the figure, in this case, the algorithm similarly made it possible to identify both the liquid level in the test tube and its contour, despite the presence of glare in the image. A quantitative analysis of the quality of recognition was performed for six real images of blood tubes (Figure 5). Figure 6a shows the dependence of the proportion of the recognized area of blood on the value r of repeatability for six images. Figure 6b shows the ratio of the number of pixels that were erroneously recognized (pixels of other objects) to the total number of recognized pixels as a function of the value of repeatability. As can be seen from the figures, with an increase in the value of repeatability, the proportion of erroneously recognized pixels decreases. However, the proportion of the recognized blood area reaches 87% with 20% of erroneously recognized pixels of other objects to the total number of recognized pixels (Figure 5c).

4. Conclusions

The proposed algorithm for determining the level and contour of blood in images based on the HSV color model and its software implementation showed good results both for images of real blood and in a real environment using a fluid that simulates blood. As part of future studies, we will analyze the effectiveness of the proposed algorithm for recognizing real blood samples in a real environment under conditions of different distribution of blood phases in test tubes after centrifugation.