Exploiting Image Processing and Artificial Intelligence Techniques for the Determination of Antimicrobial Susceptibility

Abstract

Antimicrobial susceptibility tests, achieved through the use of antibiotic-impregnated disks in a suitable laboratory environment, are conducted to determine which antibiotics are effective against the bacteria present in the body of an infected patient. The Kirby–Bauer method, a type of disk diffusion antimicrobial susceptibility test, is currently widely applied in microbiology laboratories due to its proven effectiveness. In our study, we developed an algorithm that utilizes image processing techniques to detect the inhibition zones of bacteria. A certain color depth acts as the threshold for the inhibition zone, with its radius determined according to the size of the reference object. This approach facilitates the measurement of inhibition zones and employs machine learning and deep learning to categorize antibiograms, followed by determination of whether a bacterium on the disk is sensitive or resistant to the antibiotics applied. The focus of this research is creating an automated interpretation system for antimicrobial susceptibility testing using the disk diffusion technique, thus simplifying the measurement and interpretation of inhibition zone sizes.

1. Introduction

Antimicrobial susceptibility testing (AST), conducted in a laboratory setting or under artificial conditions, aims to determine the susceptibility of bacteria from samples taken from a living organism to specific drugs and their concentrations during an infection. The agar dilution, broth microdilution, and disk diffusion methods are currently widely used in antimicrobial susceptibility tests. The disk diffusion method (DDT) is preferred in microbiology laboratories for its simplicity, repeatability, and cost-effectiveness in determining antibiotic sensitivity [1].

In the DDT method, bacteria at a specific concentration are spread on a culture medium, and disks soaked in antibiotics are placed on top. An inhibition zone (a circular area where bacteria do not proliferate) is formed around the disk. Judgments regarding the sensitivity of the microorganism are made by measuring the inhibition areas formed around the disk [1,2,3]. The measurement of zone diameters is conducted using a caliper or ruler. However, this method poses a risk of contamination for the person performing the measurement and may lead to user-related errors.

At present, antimicrobial susceptibility testing through image processing is performed in many laboratories, using biomedical devices or manual measurement. The images formed after incubation are analyzed 20–24 h after the culture is placed in the prepared setup [4]. In the field of microbiology, image processing is mostly used to modify, analyze, or improve existing images. In the reviewed studies, the detection of antibiotic disks with image processing methods was mostly performed using the OPENCV-Hough Circle Transform method. After determining the position of the antibiotics in the inhibition zone measurements, the images were subjected to pixel processing by passing them through various filters. The determination of antibiotic types was achieved through various text identification and model comparison methods [5].

While manual examinations by physicians have yielded antibiogram analyses, automated antibiogram image processing has not yet achieved routine clinical use. Further research is required to develop image processing methods associated with artificial intelligence for a combined approach utilizing both manual and automated antibiogram analyses. Our study aimed to evaluate antibiograms through implementing an efficient image processing algorithm, using a machine learning method to determine the growth inhibition zones from plate images, and identifying the types of antibiograms utilizing convolutional neural networks (CNNs), a deep learning technique. With this methodology, it is possible to precisely determine the diameters of the inhibition zones and antibiograms using image processing and classification libraries.

In this study, we minimize contamination problems through performing automatic diameter measurement according to a reference object, guaranteeing a faster and more efficient measurement process through automation of the diameter measurements. Then, we use a machine learning algorithm to determine the types of antibiograms. Thus, we aim to bring an innovative perspective for the identification of antibiotic disks, which are coded with letters or numbers to provide new data with the help of artificial intelligence.

The AST is a clinical test that detects the susceptibility of a microorganism to antibiotics. The examination of antibiogram tests is usually undertaken manually, which can lead to human error and lengthy procedures. In order to read and explicate the antibiograms, some effort has been put into the development of automated devices; however, it is often costly to operate these medical devices in different locations. Therefore, automated computerized image processing has been identified as an effective method for the detection and interpretation of antibiograms. This study aims to provide new solutions to overcome the identified problems through developing processes for the automatic reading of antibiograms, which could be much faster and more accurate than laboratory measurements. Automated measurements are expected to enable the commencement of new clinical trials and increase awareness in this research area. The remaining sections of this paper are organized as follows. Section 2 explores a range of related works, Section 3 provides an account of the image processing methods used to measure the diameter of inhibition and the application of deep learning methods to detect the types of disk antibiograms, and Section 5 comprises a summary of this study.

2. Related Works

Microbiologists can utilize antibiogram disk diffusion assays to ascertain an organism’s susceptibility to specific antibiotics [6]. Antibiograms are employed for a variety of reasons, but their most popular usage is supporting the clinical trials of novel therapies for suspected infections in a clinical setting [7]. Variations in conditions can cause antibiograms to differ in consistency when used for this purpose. To improve treatment success rates, more precise test reports are needed. Despite the fact that the medical microbiology laboratory has been the site of several investigations, the most effective way to analyze and convey data in this context has not yet been decided.

Okowat et al. aimed to develop an automated interpretation of antimicrobial susceptibility testing based on image processing [8]. In their system, they adopted the Kirby–Bauer AST disk diffusion susceptibility method. The test was able to automatically detect and quantify zones of inhibition and automatically detect the letter and text codes of antibiotic disks on the plate. One of the major advantages of automated susceptibility methodologies is the reduction of labor. Another advantage is that these systems enable faster reporting of susceptibility results and, therefore, potentially earlier initiation of appropriate antibiotic therapy. Automated antimicrobial susceptibility testing based on the disk diffusion method has the advantages of standardization, reduction in measured variation and human error, more precise results, reduced analysis and reporting time, and improved hospital information management. To estimate the direction of the antibiogram, the method of placing an ellipse on each letter was used. Then, the bubble sorting algorithm was used to estimate the rotating angle of the antibiotic disk drug. Each letter was cropped for segmentation, and then represented through binary image transformation by applying adaptive thresholding methods. As a result, only the images of antibiograms were taken with traditional OpenCV methods, and binary image transformation was performed. Only some of the phases of antibiogram type identification were implemented. In particular, the text recognition phase was not included in their study, and no information on the detection and measurement of inhibition zones was given.

Alonso et al. [9] aimed to minimize user and measurement errors by designing a software tool that semi-automated the process of capturing images of disk diffusion tests. They developed AntibiogramJ as a Java application, which included the OpenCV library, and a library applied in bioinformatics called ImageJ, providing functionality for image processing. These two libraries were combined and examined using the IJ-OpenCV library (http://joheras.github.io/IJ-OpenCV/, accessed on 15 April 2023). In addition, an embedded database provided by AntibiogramJ—the JavaDB library—was used. In this work, the reduction in manual work processes, the shortcomings of the architecture of the AntibiogramJ library, the need for an open architecture, and the increased mobility of a mobile application were noted. As AntibiogramJ does not distinguish which devices and conditions were used to capture plate images, there is variability in these images. Therefore, it is necessary to modify the quality of the images using the functionality offered by AntibiogramJ. First, the user can perform image selection manually or automatically to obtain plate images only. Automatic area selection was performed using the OpenCV library and the Hough circle transform [10]. Moreover, it is possible to adjust the brightness and contrast of the plate images both manually and automatically with the use of ImageJ. The benefits of automation offered by these systems include enhanced standardization, leading to greater accuracy, improved data management, with a simultaneous decrease in transcription errors, quicker availability of results, and reduced risk of exposure to cultivated pathogens. Archived images of culture plates have increased staff productivity, enabling more comprehensive quality reviews and the creation of real-world training sets. Moreover, it is possible to use the image library as a training tool [9].

The growth of E. coli bacteria was observed by analyzing images during 24 h of incubation [11]. Changes in the disk inhibition zones over time represented positive results in terms of preventing contamination in environments where microbiological processes are carried out manually with image processing techniques. With this method, zone measurements were carried out according to the reference disk diameter (6 mm), but no information on antibiogram types was given. In addition, Fidan et al. have noted that the study by Ferrari et al. [12], which used convolutional neural networks (CNNs) to count colonies, reported 92.8% accuracy counts. Utilizing their developed method, quantitative data were obtained, and contamination problems were prevented. However, it was emphasized that the study should be repeated for different types of bacteria and antibiotics [11].

Various AI approaches have been developed that have had success in microbiology. Lv et al. [13] briefly explained how AI could address antimicrobial resistance (AMR) through predictive modeling, smart antibiotic use, and the exploration of antimicrobial peptides and antibiotic combinations. In [14], Weis et al. systematically searched for studies that use machine learning algorithms to improve species identification and susceptibility testing based on MALDI-TOF mass spectra [15]. A total of thirty-six studies were identified that utilized various machine learning algorithms to identify species and support antimicrobial resistance profiling. There was a significant overlap in the pathogens studied and the types of machine learning algorithms used. However, a quality assessment revealed several shortcomings in current machine learning implementations, which need to be addressed before routine diagnostic use can be considered.

Pascucci et al. [16] developed an offline mobile application technology which analyzed disk diffusion ASTs and presented interpreted results to users. The methodology of this study was described as combining machine learning (ML) and traditional image processing algorithms for automated AST analysis using a mobile application. The OPENCV library from C++ libraries was used to detect the inhibition zones and the Tensor Flow library from ML libraries was used for antibiogram detection. In the experimental section, several problematic images were detected in the datasets, due to damaged plates and low contrast. The difficulty of isolating some inhibition zones, even by eye, was mentioned. In comparison, the combined effect of bacterial pigmentation and ambient illumination introduced a significant degree of variability in the bacterial inhibition intensity contrast. In the experiments, images with low contrast gave worse results. Consequently, for each of the two tests, a neural network model was trained to categorize the images as positive or negative. Accuracy rates of 99.7% and 98% were achieved with models identifying clindamycin-inducible resistance and extended-spectrum beta-lactamases (ESBLs), respectively. However, the models exhibited suboptimal performance across a diverse range of images, including variations in plate layout and bacterial texture. Given the potential seriousness of classification errors in interpreting antibiotic susceptibility testing (AST) and subsequent patient treatment, the developed methods incorporated a user confirmation step within the automatic detection of the resistance mechanism. Thus, the recommended strategy involved integrating models of the resistance mechanism into the application and training them using supervised machine learning methods to enhance the overall functionality. Consequently, the current version of the mobile application includes an automatic detection procedure grounded in machine learning, a subset of AI.

Unfortunately, AI approaches still have several limitations. Existing AI models struggle to accurately predict high-dimensional features due to data constraints, leading to reduced accuracy. To overcome this challenge, exploring the automatic annotation of unlabeled data through unsupervised learning appears promising for future research. This approach could enhance the capabilities of AI models by leveraging larger datasets without requiring extensive manual labeling.

Furthermore, current AI applications are often constrained to processing datasets with similar distributions, which highlights their insufficient generalizability. Consequently, techniques such as transfer learning and few-shot learning are expected to become increasingly applicable in addressing complex issues such as antimicrobial resistance in the long term [13].

In previous studies, challenges emerged in identifying antibiotic types due to a reliance on traditional text recognition methods, as exemplified in the study by Thitikarn Okowat et al. [8]. Our research, while not offering additional insights beyond the work of Pascucci et al. [16], identified specific challenges with low-contrast, noisy, or unevenly distributed inhibition zone images, resulting in ineffective diameter measurements, consistent with the existing literature.

To address these challenges, we advocate for refining clustering methods using artificial intelligence techniques to enhance the inhibition zone measurement accuracy. Specifically, our study highlights the shortcomings of the K-Means algorithm in detecting pixel grayscale transitions within certain low-contrast and noisy inhibition zone images, leading to inaccuracies in measurements. This underscores the ongoing need for the exploration and application of advanced artificial intelligence methods in this field, with the aim of enriching the research available and inspiring further investigation among researchers.

3. Materials and Methods

Various methods are employed for antibiogram tests, with the disk diffusion method (Kirby–Bauer method) being a significant example. This method offers the advantage of easy interpretation, although it has slightly lower sensitivity compared to other methods. In this approach, antibiotic-impregnated disks are placed at specific doses, and sensitivity or resistance is determined by examining the zone around the disk after incubation.

To test antibiotic sensitivity, a suitable solid culture medium is prepared and inoculated with a turbid liquid microbial culture. The process ensures an even distribution across the Petri dish. Antibiotic disks are then placed, and the dishes are incubated at the required temperature and time. After incubation, the inhibition zones around the disks are measured in millimeters and compared to standard values to determine sensitivity or resistance.

In this section, we describe the image processing methods used to measure the diameter of the inhibition zone, as well as the application of deep learning methods to detect the types of disk antibiograms. Python3 and some libraries of Python3 were used as the programming language. These libraries included pandas, cv2 (open cv), os, numpy, matplotlib, tensorflow, and keras. Online platforms such as roboflow, Google Colab, and GitHUB were also utilized.

3.1. Detection of Antibiotics on the Culture Disk

It is possible to extract features from antibiogram Petri dish images through exploiting information derived from computerized image processing. Consequently, culture disk analysis makes extensive use of images in the tiff, jpeg, png, and bmp formats. The dataset used to identify antibiotics on the culture disks was obtained from Marco Pascucc’s “AST-image-processing” project, which was made public on GitHUB [17].

In this study, there were five image analysis steps: inputting data, image preprocessing, filtering, reading the regions, extracting the original digital features, and analyzing and determining the boundaries. The processing steps, including the image processing sections through the OPENCV open-source library, were as follows:

• All the accessed original antibiogram plaque images were saved in JPEG format. Image sizes of 3096 × 4128 × 3 were given as input to the program. These image sizes were from a fixed camera and the background color was a dark fixed background, enabling the algorithm to achieve better results. An example of the culture disk is shown in RGB format in Figure 1.
• RGB raw images were given in a 3D matrix. There was no need to use a color-related property for the antibiogram images. Thus, all raw original antibiogram images were transformed to grayscale images. Any details were removed in order to extract the numerical features relevant to the antibiogram images, other than the Petri dishes, since Petri dish images were the main focus of our processing.
• A median filtering technique approach was applied to remove noise from the images. This type of noise removal is a traditional pre-processing step used for better results in post-processing. The median filter aimed to remove the small-scale image contamination present in the images [18].
• The Circular Hough Transform (CHT) method was employed to detect the circles. CHT is commonly used in conjunction with its library in OpenCV [19]. To detect the circles, we required the specific parameters of a circle, which in the circle equation (x − x₀)² + (y − y₀)² = r² consist of the parameters x₀, y₀, and the radii representing the center. To find the sample image of the object in the positive class, circular shape centers were detected by means of the Circular Hough Transform. This method was applied to the images of the circles to be found, indicating the range of radii. In an image with a resolution of 3096 × 4128 pixels, the fixed diameter of the antibiotic disks was taken as 6 mm. This meant that setting the radius as a parameter ranging from 50 to 80 was a more efficient method. The “Sensitivity” feature was chosen as 0.95. These parameters were set for the detection of the antibiogram images after experimentation. After applying the Circular Hough Transform, three outputs were obtained: the matrix of the centers of the circles with the x and y coordinates (called “centers”), the radius matrix of the circles, and the metric showing how close the circle was. Then, the circles in Figure 2 were drawn on the circular images, represented in green, using the “centers,” “radii,” and “EdgeColor” parameters. In addition, the circled antibiotics were represented in blue and numbered (Figure 2). These procedures made it possible to accurately identify the antibiotics on the disk.
• For each x, y, and r (radius) value, a frame image of the antibiotic disk was taken and provided as input to another function. These cut antibiotic images (Figure 3) were presented as input to the machine learning layer. Here, the antibiotic type was detected.

First, for the x, y, and r values in the sample images, the formula (x − r, y − r, x + r, y + r, x + r) was used to find the coordinates X1, Y1 by subtracting r from the x, y values of the circled images, and X2, Y2 by adding r. These four coordinates were obtained by using the OPENCV image cutting function to obtain the images, as shown in Figure 3.

The statistical results of this study with the Hough circle method are shown in

Table 1. Antibiotic disk detection rates of the test plates.

Input Image	Method Applied	Number of Antibiotics on the Petri Dish	Number of Antibiotics Detected Percentage of Success	Percentage of Success
test_1.jpg	Hough circle	16	16	%100
test_2.jpg	Hough circle	13	13	%100
test_3.jpg	Hough circle	13	13	%100
test_4.jpg	Hough circle	16	16	%100
test_5.jpg	Hough circle	16	16	%100
test_6.jpg	Hough circle	5	5	%100
Total Percent Success:				%100

. Then, the antibiotic disk images could be compared using CNNs, and antibiotic type prediction was carried out in the last phase of our study.

3.2. Measurement of Inhibition Zone Diameters on the Culture Disk

Median blur has been shown to be an effective method for removing small noise. Using the median blur method, we successfully minimized small noise in our study. The comparison of the sample image with the median blur (17) filter is shown in Figure 4.

In order to detect the inhibition zones, the median blur feature was used to remove some of the small particle noise around the antibiotic disks from the image samples in our culture dish, as shown in Figure 4. The coordinates and radii of the previously detected antibiotic disks on the image are given in Section 3.1 in Step 4. Here, keeping the Y-axis fixed and taking the coordinates as input, all 8-bit (0–255) pixel values between the X1–X2 coordinates were taken as a maximum of 3 × R (i.e., 6 × r of the antibiotic disk). This value was translated by X1 + r in order to start from the outer arc of the antibiogram disk at position X1.

We fixed the pixel cluster with a predefined parameter cluster of two. Through two clusters, it effectively classified the inhibition zones. The K-Means algorithm operates in iterative fashion, assigning each data point to one of the K groups according to specified parameters. Clustering of data points occurs by grouping them based on similarities in features.

Thus, by keeping the Y1 coordinate constant and moving linearly through the X[n1 … n2] coordinates, the antibiogram detected the region changes with different numerical values in the areas outside the disk. This cluster change of pixel values was as follows:

[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

With the K-Means algorithm, the change point of the dataset clustered as two was calculated. By taking the index information of this calculated value through a loop, the change point of the inhibition zone diameter was calculated in pixels. Then, the X1, X2, and index values of this antibiotic disk were taken as r, and the zone of inhibition was determined by drawing a circle with these parameters. The fixed size of 6 mm of an antibiotic disk with r = 55 px was taken as a reference. For the example with an inhibition zone value of 49 px, the calculation was as follows:

Radius = [(Reference object + Detected zone)/Reference object] × 6 mm = [(55 + 49)/55] × 6 mm = 11.345 mm.

In this section, the method of measuring the inhibition zones is explained. Figure 5 shows the circled zones of inhibition. The 8-bit grayscale color distributions of (f) test_6.jpg image are given in Figure 6.

When reading these inhibition zone regions, the desired result could not be obtained for low-contrast materials in regions where there was no uniform and symmetrical distribution, and in zone images that went beyond the plate. These measurements are also given in

Table 2. Inhibition zone measurements evaluations.

Input Image	Pixel Algorithm Applied	Number of Inhibition Zones	Number of Correctly Detected Inhibition Zones	Success Percentage
test_1.jpg	K-Means	16	14	87.5%
test_2.jpg	K-Means	13	11	84.6%
test_3.jpg	K-Means	13	13	100%
test_4.jpg	K-Means	16	15	93.7%
test_5.jpg	K-Means	16	16	100%
test_6.jpg	K-Means	5	3	60%
Total Success Rate:				87.6%

. In the background of Test_6.jpg, the circular zone RGB values exhibiting weak contrast and a lack of uniform distribution in the frictional part of the disk led to misleading results when converted to grayscale. This can deceive the K-Means pixel clustering algorithm, resulting in inaccurate outcomes. To overcome this challenge, images in regions with low-contrast inhibition zones and irregularly distributed feeding sites can be trained using artificial intelligence methods, in order to develop more accurate clustering algorithms.

Comparing Manual Measurement with the Proposed Method

At present, antibiotic susceptibility tests (ASTs) are performed in many laboratories using biomedical devices or through manual measurement. In traditional methods, the zone of inhibition is manually measured using a ruler, which can lead to measurement errors. In many microbiology laboratories today, measurements are still conducted manually with a ruler. This study automated the measurement of the inhibition zone diameter on a 1600 × 1200 px disk, as shown in Figure 7. Both manual measurement (with a ruler) and automatic measurement using the method proposed in this study are shown.

In Figure 7, it can be observed that a disk, captured at a resolution of 1600 × 1200 px (test_reel.jpg), was measured with a ruler. The process involves identifying an antibiotic, determining its center coordinates (x, y), and then using the K-Means algorithm to analyze pixel changes in the inhibition zone (Figure 8).

Diameter Calculation: The OpenCV Hough Circle method was used to identify the antibiotic. When the antibiotic was detected, three pieces of information were obtained: the x and y coordinates, and its radius (marked by the green circle and number 1). The coordinates of the identified antibiotic were X: 806, Y: 704, r: 55. In this example, a 56-pixel radius represented the size of the antibiotic disk in the 1600 × 1200 pixel resolution image.

Calculating the inhibition zone: The K-Means clustering algorithm indicated that the end of the gray color change (in the inhibition zone) occurred at 1102. pixel, shown in Figure 8. (at 1102 × 204 indicated by blue and red lines); the r inhibition zone is 1102 px − 806 px = 296 px. The inhibition zone’s diameter [mm] is (r inhibition zone)/(r antibiotic) × 6 mm, which results in (296)/(56) × 6 mm = 31.714 mm.

3.3. Disk Antibiotic Image Dataset

In this study, antibiotic-impregnated disks and their images were downloaded from an open-source website (https://github.com/mpascucci/AST-image-processing/tree/master, accessed on 15 April 2023). The downloaded AST set A1 and A2 data were collected routinely in the microbiology laboratory of the University Hospital in Creteil, France. Samples were taken from patients at the hospital. The preparation and analysis of AST was not primarily designed for this study; instead, normal hospital procedures were followed. The AST set A3 consisted of eight Petri dishes prepared at Medecins Sans Frontieres Hospital in Amman, Jordan. The plates of this set were inoculated with microorganisms provided by the American Type Culture Collection, which are regularly used for quality control. These strains are among the most commonly utilized pathogens and have known diameters of inhibition against several antibiotics. Pascucci et al. defined all data acquisition processes and methods in [16].

Our dataset consisted of 9770 images of 23 different antibiotic classes in total. An example of the antibiotic disk image dataset is given in Figure 9. The dataset was taken, and the augmentation process was applied on the Roboflow online platform, yielding antibiotic disk images of 23 different classes. The class names of the antibiotics used were as follows: [‘AK30’, ‘AMP10’, ‘ATM30’, ‘AUG30’, ‘C30’, ‘CAZ10’, ‘CD2’, ‘CIP5’, ‘CN10’, ‘CRO30’, ‘CTX5’, ‘E15’, ‘ETP10’, ‘FC10’, ‘FEP30’, ‘FOX30’, ‘NOR10’, ‘P1’, ‘RD5’, ‘SXT25’, ‘TC75’, ‘TOB10’, ‘TZP36’].

3.4. Deep Learning and Convolutional Neural Network Model

Deep learning is an extended subset of the machine learning (ML) family and provides more efficient ways to implement ML. The “deep” part of deep learning is a technical term, which refers to the number of layers or sections in the “network” part of “neural networks.” Deep learning has played a critical role in the development of highly automated systems, such as autonomous vehicles and natural language recognition and understanding. Convolutional neural networks [20] are one of the most widely known and used types of deep learning models. They have achieved significant success and innovation in computer vision and image processing. The basic architecture of a CNN is shown in Figure 10. A CNN consists of multiple layers, including the input layer, convolutional layer, pooling layer, and fully connected layer. The input layer’s role is to take the pixel values of the image as input. The next convolutional layer is responsible for producing output based on kernels or specific filter values. The output obtained from a convolution operation and the pooling layer are used to reduce the representation size and accelerate computation. In the fully connected layer, the image—which has gone through several convolutional layers and pooling layers and is in matrix form—is flattened into a single vector.

In our study, we used the “Roboflow” online platform to develop a deep learning model for our dataset of 9770 images [21]. The dataset was also subjected to pre-processing for color depths and augmentation. These processes were applied as an auto-orientation of pixel data (with EXIF-orientation stripping) and grayscale (CRT phosphor), respectively.

The data, belonging to 23 classes in our dataset, were read from a file and classified by means of sub-file divisions. Of the 9770 images classified, 1954 images, corresponding to 20% of the data, were used as test data. As the images in the datasets were approximately 120 × 120 pixels in size, the height of the image was taken as 150 and the width as 150. A value of 32 was given as the batch value. A label dictionary was created to retrieve class names against the label indices used to train the model. After the training dataset preparation stages were completed, a CNN model was created to train the image datasets. The first layer of our CNN model was defined as the input layer. The dimensions of the input data were parametrically input to the system with the relevant input command set of the Keras library. The size input of our dataset was set to 120 × 120 with the input shape parameter. The “sequential” model was used in model selection. This model is defined as a class object in artificial neural networks. The “add” command allowed us to sequentially add model layers to this “sequential” model. Our first layer was the Conv2D filter layer. This layer was configured with a kernel size of (5, 5) and a total of 128 filtering parameters. The input size was parametrically defined as the pixel value equivalent to the width and length of our dataset. The second layer, following the convolutional layer, was the activation (“relu”) layer. The rationale behind selecting this layer was that RELU activation functions, which are only active for positive values, do not activate a potential negative output in the intermediate layers of the neural network. Consequently, it will not be able to designate all neurons as active simultaneously. This is effective in terms of performance, enabling the system to operate faster and more efficiently. Subsequently, MaxPooling2D (2 × 2) was utilized with the pool size parameter to diminish the spatial dimensions of the output volume. The pooling layer transformed the matrix inputs from the activation (‘relu’) layer into 2 × 2 matrices with max pooling. These matrices were maximized and output to the subsequent layer. Then, owing to batch normalization, the layers in the network did not have to wait for the preceding layer to learn, facilitating simultaneous learning. This accelerated our training. These layer structures were reiterated, commencing with the Conv2D filters = 64, kernel size = 3 × 3 option. At the subsequent level, the steps were repeated with Conv2D filters = 32, kernel size = 3 × 3. Finally, a fully connected layer was designed, and the flatten layer was applied. The role of this layer is to prepare the input data for the last and most crucial fully connected layer. This layer is tasked with converting the matrix from the other layers into one dimension. Dropout (0.5) was applied to the output of the flatten layer. This layer was utilized after the fully connected layers. In our model, dropout was used to sever the connections in the fully connected layers. Thus, the nodes possessed less information about each other. Consequently, the nodes were less influenced by each other’s weight alterations. Therefore, we achieved a more consistent model with the dropout method. The last layer in our model was dense (units = 23, activation = “softmax”). Since we had 23 classification categories, we set this parameter to efficiently predict these classes in the last layer. We also used the softmax function to perform the classification tasks. Before compiling our model, we set the optimize parameter to “adam” in the optimize option. The visualization and the summary of our sequential model are illustrated in Figure 11 and Figure 12, respectively.

3.5. Transfer Learning CNN Model

Transfer learning, a technique in machine learning, involves repurposing or fine-tuning a model trained on one task for another related task. In convolutional neural networks (CNNs), this method utilizes a pre-trained CNN as a starting point for tasks like image classification, object detection, or image segmentation. The underlying concept is that the pre-trained CNN already comprehends relevant features, such as edges, textures, and shapes, crucial to the new task. Leveraging these pre-trained weights accelerates learning on the new task, requiring fewer training examples compared to training a new model from scratch. Our study employed the same CNN model illustrated in Figure 11 and Figure 12, except for the initial layer. Instead of the first layer, we utilized the DenseNet201 and VGG16 models as base models. DenseNet201, a variant within the DenseNet family, is a deep neural network architecture introduced by Huang et al. in 2017 [22]. This specific configuration comprises 201 layers and has been trained on the ImageNet dataset, designed for efficient image classification tasks. It incorporates 20,242,984 pre-trained parameters. VGG16, introduced by Karen Simonyan and Andrew Zisserman in 2014, is a widely adopted deep neural network architecture for image classification [23]. Comprising twenty convolutional layers and three fully connected layers, it extracts features from input images through convolutional layers and conducts classification using fully connected layers. This network employs a total of 138,357,544 parameters, making it relatively larger than other contemporary models for computer vision tasks.

4. Results and Discussion

To determine the success of the models following training, their performance in the assigned tasks was assessed by evaluating quantitative performance metrics including the accuracy, precision, sensitivity, F1-score, and loss. Additionally, the duration of the training sessions was subjected to analysis.

$$Accuracy=\frac{TP+TN}{TP+TN+FP+FN}\times 100$$

(1)

$$Precision=\frac{TP}{TP+FP}\times 100$$

(2)

$$Recall=\frac{TP}{TP+FN}\times 100$$

(3)

$$F1=\frac{TP}{TP+1/2(FP+FN)}\times 100$$

(4)

The definition of parameters used in (1), (2), (3), and (4) are given as follows:

TP (True Positive): Correctly identifying something as true. When a known true value is correctly labeled as true.

FP (False Positive): Incorrectly identifying something as true. When a known false value is incorrectly labeled as true.

TN (True Negative): Correctly identifying something as false. When a known false value is correctly labeled as false.

FN (False Negative): Incorrectly identifying something as false. When a known true value is incorrectly labeled as false. The size of the confusion matrix depends on the number of classes in the examples being considered.

The value obtained from the loss function measures how different the model’s predictions are from the actual values. High loss values suggest inaccurate predictions, while low values indicate accurate predictions.

We used the N-fold cross-validation method to score our CNN model’s success. We chose N = 5, which means we split the training set into five randomly selected equal parts and used each part N-1 times for training and one time for validation. The 5-fold method allowed us to test our CNN against overfitting and selection bias. Our model was fitted with 75 epochs (cycles) in each fold, and the best results obtained are shown in Figure 13. When the Training score: Epochs graph in Figure 13a is examined, it can be seen that our model learned regularly as the number of epochs increased. The training score, expressed by the blue dotted line, increased continuously, and became horizontal as it approached 75 epochs. The validation score, represented by the orange dotted lines, was significantly different from the training score at times. This was due to the inconsistent number of datapoints in the antibiogram disk classes in our dataset. To avoid this inconsistency, it would be appropriate to remove some classes from the dataset. After the 50th cycle, the training score and the validation score remained proportional.

The error matrix in Figure 14 shows quite large values in the classification rates of our classes. This was because these data were based on our best resulting fold. To determine the real success rate of our model, we summarized all five cross-validation results; statistical information from this is given in

Table 3. Average values of the CNN model’s 5-fold cross-validation.

Classes	Precision	Recall	f1-Score	Support
AK30	0.992	0.958	0.976	106
AMP10	0.966	0.968	0.966	54
AMP2	0.992	0.96	0.974	66
ATM30	0.962	0.992	0.978	72
Aug-30	0.776	0.86	0.818	42
C30	0.946	0.968	0.954	67
CAZ10	0.784	0.94	0.854	29
CD2	0.916	0.906	0.91	100
CIP5	0.978	0.912	0.946	103
CN10	0.952	1	0.976	62
CN30	0.926	0.892	0.91	58
CRO30	0.994	0.988	0.994	38
CTX5	0.922	0.974	0.946	63
E15	0.85	0.898	0.876	33
ETP10	0.856	0.966	0.906	72
FC10	0.938	0.862	0.898	90
FEP30	0.824	0.556	0.654	14
FOX30	0.936	1	0.968	23
IMI10	0.73	0.634	0.68	40
LEV5	0.93	0.862	0.896	96
MRP10	0.884	0.846	0.864	47
NOR10	0.91	0.966	0.938	110
P1	0.856	0.84	0.848	45
Accuracy			0.92	1430
Macro avg.	0.90	0.90	0.90	1430
Weighted avg.	0.92	0.92	0.92	1430

.

We trained the VGG16 and DenseNet201 models on the same dataset with 100 epochs each. The other parameters were the same as for our CNN model.

Table 4. Success rate of DenseNet201.

	Precision	Recall	f1-Score	Support
Accuracy			0.89	1430
Macro avg.	0.88	0.89	0.89	1430
Weighted avg.	0.90	0.89	0.89	1430

and

Table 5. Success rate of VGG16.

	Precision	Recall	f1-Score	Support
Accuracy			0.91	1430
Macro avg.	0.90	0.90	0.90	1430
Weighted avg.	0.92	0.91	0.91	1430

show the results of both transfer learning models. The DenseNet201 model had a 0.89 success rate, and VGG16 had a 0.91 success rate. Our CNN model worked slightly better on averaged data.

The experimental studies and visualizations indicated that higher success rates can be achieved in the field of classification of antibiotic disks coded with different shapes and codes through using the correct model approach, the correct number of epochs, and avoiding overfitting. Classification success was achieved by our CNN model with a precision rate of 0.9, an f1 score of 0.9, and an accuracy rate of 0.92 for the 23 classes in our dataset.

5. Conclusions

In test applications of antibiograms, human error can occur when measuring the antibiotic inhibition zone. Despite this, manual measurement methods are still in use. This study aimed to enhance the efficacy of antibiogram susceptibility test measurements through minimizing human factor errors occurring during these measurements.

In this study, inhibition zone measurements of the antibiogram susceptibility test were performed by means of automatic identification methods and an artificial intelligence-based clustering model. The deep learning CNN model was then used to classify the antibiotic types. The first of the three steps of our study involved determining the threshold value within a certain coefficient range used for identifying antibiotics, and successfully detecting the antibiotic disk image using the “Hough Circle” method, due to its high contrast value relative to the Petri dish. The antibiotic disks in six test Petri dishes were fully detected. In this step, the success of our method was 100%. Fidan et al. utilized antibiotic disk diameters of 6 mm in their studies and indicated that this corresponded to 55 pixels using pixel measurement techniques. In their research, they reported detecting antibiotics by passing them through different filters, including a blurring filter [11].

In step 2, despite utilizing various noise removal methods in the zone measurement, some zones were not accurately detected. Pascucci et al. discussed the challenges of detecting zones in digital images with damaged plates and extremely poor contrast between bacteria and the inhibition zone [16]. They developed and trained two machine learning models for identifying resistance mechanisms in images. While the accuracy results were promising, they were concerned about overfitting due to the small size of their training datasets, especially for the mobile application’s requirements. However, the integrated Expert System managed these situations by prompting the user to confirm or exclude the presence of the shapes when they were likely to occur. Their application’s fully automated measurement procedure demonstrates a 90% agreement in susceptibility categorization compared to a hospital-standard automatic system. Fidan et al. described the preliminary steps of image processing through a closed setup in zone measurements and suggested that problems encountered in image preprocessing can be significantly reduced by utilizing the homogeneous illumination technique [11]. Additionally, they found that image quality improved without glare and reflection when the light intensity was homogeneously distributed in a closed environment using plexiglass and vinyl material. In our study, the success of the diameter calculation of inhibition zones was measured at 87.6%. Better results can be achieved through adopting a combined method with different parameters, such as lighting, different coloring of the Petri dish background, and image processing filters with different parameters when measuring inhibition zones. Future studies should consider that different combined methods could contribute to the literature.

The final phase of our study was conducted using the CNN machine learning technique. The “amman_atb_data” dataset, consisting of 9770 images and 23 classes, was utilized. With this dataset, a 92% antibiotic disk classification success was achieved. The dataset could be organized more effectively by gathering fewer images of some classes. Moreover, performing more image augmentation could further enhance the success rate.

Antibiogram susceptibility tests have been automated, and commercial applications have been realized in this field; Biomik V3 (US-based) [24] and VITEC 2 (US-based) [25] are the most widely used at present. The advantages of these devices include improved standardization, as well as greater accuracy and fewer errors. However, these computerized automated systems are too costly for clinical laboratories.

Furthermore, in many countries, legal procedures and permissions are required for the practical implementation of automatic interpretation systems in microbiology laboratories. While standardizing testing procedures can enhance quality and efficiency, they remain impractical for settings with limited resources due to cost and infrastructure requirements. Additionally, performing such a study requires both human resources and time with a software team. Future studies will contribute to the current study through further developing inhibition zone measurement techniques and finding a more effective method for low-contrast inhibition zones. This study may be the beginning of the development of device-based domestic antibiogram susceptibility tests.