An Enhanced Deep Learning-Based Pharmaceutical Crystal Detection with Regional Filtering

Abstract

In the process of pharmaceutical crystallization, the automatic detection of crystal shapes in images is important since controlling the morphology of the crystals improves the quality of pharmaceutical crystals. In this paper, a novel image detection method called RECDet is proposed. It leverages an automatically adapted binary image to bypass background regions, thereby reducing the detection field. In addition, the method greatly reduces the training time while improving the detection accuracy by using a specially designed detection box for the crystal shape. The performance of our model is evaluated through experimental analysis on a publicly available glutamate crystal dataset and a self-made entecavir pharmaceutical crystal dataset. Experimental results show that RECDet improves the accuracy of prediction bounding boxes by more than 2% compared to other popular models and achieves a classification accuracy of 98%. It can be used as a promising tool in the application of pharmaceutical crystallization control.

1. Introduction

Control of pharmaceutical crystallization is a pivotal step in pharmaceutical development and manufacturing, involving the regulation of crystal size, shape, and purity to ensure optimal pharmaceutical efficacy and bioavailability [1,2]. Traditional crystallization control methods that rely on manual monitoring are time-consuming, labor-intensive, and often unreliable. Recent advancements in deep learning have significantly enhanced image analysis, fostering the development of models that improve the accuracy and efficiency of object detection [3,4,5]. As image recognition technologies evolve, there is increasing interest in automating the monitoring of crystallization processes [6,7,8,9]. Nevertheless, the extensive application of image recognition in crystallography is still not fully explored [10,11,12,13,14]. In contrast to specialized models used in medical image recognition, crystallography has predominantly employed the general image processing frameworks [15,16].

For example, Gao et al. [17] applied Mask R-CNN in the field of crystal recognition to record the transformation process of glutamate crystal morphology, including quantity, size, and surface area. Huo et al. [18] employed the UNet for crystal segmentation, which allowed for a detailed analysis of the particle size distribution during the crystal growth stages. Wu et al. [19] analyzed crystal images using S2A-Net in the field of aerial imaging and measured the two-dimensional size of crystals using a minimum-fitting rectangle. Manee et al. [20] utilized a single-stage segmentation model RetinaNet to segment crystals for real-time detection of crystal size distribution.

Despite these successes, the general object detection frameworks applied to crystal shape or type prediction still encounter numerous challenges, such as the limited dataset and algorithmic accuracy. These frameworks often focus excessively on image details to maintain detection effectiveness across various scenarios, leading to excessive attention to background impurities or water disturbances, which may hinder detection efficiency and accuracy [21,22,23]. Consequently, extensive data are required to filter out these distractions for model learning.

This study introduces a novel object detection framework named RECDet, which is expressly crafted for crystal image detection by employing background region filtering. It algorithmically generates a binary image to separate the foreground and background regions, and then inputs this information into a neural network, bypassing the background region detection to improve the crystal detection accuracy and processing speed. Additionally, considering that crystals typically exhibit uniform sizes, a specialized candidate box has been designed inspired by a two-stage object detection process, which can boost the model performance on small datasets [24]. To evaluate the effectiveness of RECDet, a public dataset consisting of 140 training images and 60 validation images is used for the test. Besides, a dataset with 180 training images, 40 validation images, and 20 test images, compiled in the COCO annotation format [25], is constructed by our experiment. The validation results of these datasets indicate that RECDet significantly improves the detection and classification of crystals, providing an effective tool for further analysis of crystal size distribution and crystallization quality.

2. Method

The detailed description of our RECDet is shown in Figure 1. We first collected crystal images and manually annotated the position of the crystals on these images. After the image acquisition, the original data was preprocessed and augmented. Then, we set the threshold and applied the binary segmentation to segment crystal regions and obtain binary images of crystals. Subsequently, a two-stage object detection model was used to modify the candidate box prediction by combining the binary crystal image. The background region can be directly skipped during the prediction process; this results in satisfactory experimental results that guarantee the identification of the crystal type and its specific position on the image.

2.1. Threshold Segmentation

As images with varying crystal densities exhibit distinct brightness levels, determining a single threshold that can perfectly segment all images is infeasible. To minimize the workload of subsequent processing steps, our RECDet model employs automatic threshold segmentation. Otsu threshold segmentation [26] is an adaptive threshold image segmentation method that uses a threshold to divide an image into foreground and background regions. The crystal image is firstly converted into a grayscale image, with a gray level interval set as [0, m]. The threshold t is then calculated and the image is divided according to two parts of [0, t] and [t + 1, m]. The calculation of threshold t is

$$F\left(t\right)=w_0\times {\left(u_0-u\right)}^2+w_1\times {\left(u_1-u\right)}^2$$

(1)

where $w_0$ is the proportion of foreground in the image with a mean of $u_0$, $w_1$ is the proportion of background in the image with a mean of $u_1$, and $u$ is the overall mean of the image calculated as $u=w_0\times u_0+w_1\times u_1$. When F(t) reaches its maximum value, The threshold t is referred to as the optimal threshold. The image is finally divided into the foreground and background regions using this threshold. The effect of the application of the above method is shown in Figure 2. It can be seen that the binary image has a very blurry division of the crystal, and the model needs to be further processed.

For crystal images under complex conditions, threshold segmentation cannot accurately distinguish the crystal region, which is often affected by the brightness of the background region and impurities in the solution. Our model is based on the results of threshold segmentation, assisted by neural network image technology, to obtain more accurate target detection results.

2.2. Network Model Structure

As shown in Figure 3, RECDet utilizes ResNet50 as its feature extraction backbone [27,28]. The input image (H × W) is divided into 5 stages. A feature map of 2048 channels is output after 49 convolution layers, and the size is recorded as M × N. The convolution process is shown in

Table 1. The specific operation process of convolution in feature extraction network.

Stage	Method
1	7 × 7, 64, s = 2; 3 × 3 max pooling, s = 2
2	{[1 × 1, 64]; [3 × 3, 64]; [1 × 1, 256]} × 3
3	{[1 × 1, 128]; [3 × 3, 128]; [1 × 1, 512]} × 4
4	{[1 × 1, 256]; [3 × 3, 256]; [1 × 1, 1024]} × 6
5	{[1 × 1, 512]; [3 × 3, 512]; [1 × 1, 2048]} × 3

.

The convolution process involves the sliding of small, trainable convolutional kernels over the input image, extracting features through the computation of the dot product between the convolutional kernels and local regions of the input data. These kernels are capable of capturing spatial and temporal patterns within the data, such as edges or textures, to form feature maps. This process is illustrated in Figure 4.

After each convolution layer, a batch normalization layer and a ReLU activation layer are connected, and the gradient stability is kept by the residual connection. By the batch normalization, the input $X_i$ is distributed to a smaller range, which is conducive to the iterative optimization of the function. Its output $\widehat{X_i}$ is expressed as

$$\left\{\begin{array}{c}\widehat{X_i}={\displaystyle \frac{X_i-\mu }{\sqrt{{\sigma }^2+ϵ}}}\\ \sigma ={\displaystyle \frac{1}{m}}{\displaystyle {\displaystyle \sum }_{i=1}^m}{\left(X_i-\mu \right)}^2\\ \mu ={\displaystyle \frac{1}{m}}{\displaystyle {\displaystyle \sum }_{i=1}^m}{X}_i\end{array}\right.$$

(2)

After the batch normalization, ReLU introduces the nonlinear factors to enhance the ability of neural networks to solve the linear fitting problem of neural networks. It is expressed as

$$F\left(x\right)=\mathrm{Max}\left(0,x\right)$$

(3)

For the feature maps derived from the backbone network of the raw image, 9 anchors are set at each pixel point, the schematic of anchor generation is shown in Figure 5.

These anchors are designed with respect to length-to-width ratios of [1:1, 1:1.5, 1.5:1] and the size scales of [8,12,16]. The aspect ratios and size scales can be adjusted according to the specific shape of the target crystal, as we have found that the size and shape of pharmaceutical crystals are always similar. Each anchor is assigned by a binary label flag corresponding to the foreground and background regions, respectively. It is differentiated using the binary images generated in Section 2.1, which is represented as

$$\left\{\begin{array}{c} flag=true, b_i=1\\ flag=false, b_i=0\end{array}\right.$$

(4)

where $b_i$ is the value at the pixel position corresponding to the anchor in the binary image. Thus, the crystal area is preliminarily determined by the binarization. Then it is judged whether the anchor corresponding to the current pixel point needs to be fed into the neural network for computation.

In the model training process, each anchor is convolved with a 1 × 1 convolution, with a channel number of 2 × 9. Thus, each anchor obtains two corresponding values, which are then used in a softmax function to perform a binary classification task, calculating the probability of the anchor being either a foreground or background class. This is represented as

$$Softmax\left(z_i\right)={\displaystyle \frac{e^{z_c}}{{\sum }_{c=1}^c{e}^{z_c}}}$$

(5)

where $z_c$ is the input data and c is set as 2 for a binary classification task. $z_i$ is the output probability value, converting the output to 1 for the foreground and 0 for the background, which is represented as

$$\left\{\begin{array}{c}{z}_i=1, z_i\ge 0.5\\ z_i=0, z_i<0.5\end{array}\right.$$

(6)

Then, a process is further refined by applying a weighted operation with the previously approximated foreground and background values:

$$b_i=\lambda b_i+\left(1-\lambda \right)z_i$$

(7)

where λ is a variable, starting with an initial value of 1. During the training process, it gradually decreases with each training epoch until it reaches 0 in the final epoch. Simultaneously, labels are continually assigned to the anchors using Equation (4) during the training. For anchors that are set to true by Equation (4), they undergo a 1 × 1 convolution with a channel number of 4 × 9. Consequently, each anchor obtains four values, which are then utilized for the box regression to refine the anchor box, thereby forming a more accurate bounding box. The calculation formula is represented as

$$\left\{\begin{array}{c}{t}_{\mathrm{x}}={\displaystyle \frac{x-x_a}{w_a}}\\ t_y={\displaystyle \frac{y-y_a}{h_a}}\end{array}\right. \left\{\begin{array}{c}{t}_{\mathrm{w}}=\log \left({\displaystyle \frac{w}{w_a}}\right)\\ t_h=\log \left({\displaystyle \frac{h}{h_a}}\right)\end{array}\right.$$

(8)

where $x_a$ and $y_a$ are the coordinates of the center point of the anchor box, and $w_a$, $h_a$ denote its width and height, respectively. $x$, $y$ are the coordinates of the center point of the prediction box generated by the neural network, and $w$, $h$ refer to its width and height, respectively. The obtained $t_x$, $t_y$, $t_w$, $t_h$ represent the offsets predicted by the neural network. The corrected prediction box is expressed as $x_a^{\ast }$, $y_a^{\ast }$, $w_a^{\ast }$, $h_a^{\ast }$, which is calculated as

$$\left\{\begin{array}{c}{x}_a^{\ast }=x_a+t_x\times w_a\\ y_a^{\ast }=y_a+t_y\times h_a\end{array}\right. , \left\{\begin{array}{c}{w}_a^{\ast }=w_a\times \exp \left(t_w\right)\\ h_a^{\ast }=h_a\times \exp \left(t_h\right)\end{array}\right.$$

(9)

If the intersection over union (IOU) between the predicted bounding box and the ground truth box is greater than 0.7 [29], it is marked as a positive sample. Conversely, if the IOU between the predicted bounding box and the ground truth box is less than 0.3, it is marked as a negative sample. Any remaining predicted bounding boxes and those that exceed the image boundaries do not participate in the final calculation. The formula for calculating IOU is

$$IOU={\displaystyle \frac{A\cap B}{A+B-\left(A\cap B\right)}}$$

(10)

where A is the area of the predicted bounding box and B is the area of the ground truth box.

2.3. Loss Function

During the neural network training process, predicted bounding boxes are scaled down by a factor of H/M to align with the feature map dimensions produced by the convolutional neural network. These adjusted regions undergo the region of interest (ROI) pooling [30], which involves the division with a 7 × 7 grid where max pooling is executed on each grid cell. The pooled output is then directed through the fully connected layers, leading to results for classification and regression tasks. The loss function for these tasks is expressed as

$$L\left(\left\{p_i\right\},\left\{t_i\right\}\right)={\displaystyle \frac{1}{N_{cls}}}{\displaystyle \sum }_i{L}_{cls}\left(p_i,p_i^{\ast }\right)+\lambda {\displaystyle \frac{1}{N_{reg}}}{\displaystyle \sum }_i{p}_i^{\ast }{L}_{reg}\left(t_i,t_i^{\ast }\right)$$

(11)

where the classification loss $L_{cls}$ uses the cross-entropy, $p_i^{\ast }$ is the true label and $p_i$ is the predicted value. The regression loss $p_i$ applies the smooth L1 loss [31], where $t_i^{\ast }$ is the true label for the bounding box parameters and $t_i$ is the predicted value.

3. Experimental and Discussion

3.1. Dataset

To validate the effectiveness of RECDet, a public dataset is first utilized, consisting of 200 images of α and β glutamate crystals. This dataset is converted to the COCO label standard for the model training and evaluation. And the dataset is shuffled and segmented into training and validation sets with a 7:3 distribution.

In addition, a new specialized dataset is created by us based on microscopic photographs of entecavir pharmaceutical crystals obtained through cooling crystallization in an ethanol-aqueous solution. Images of lower quality are excluded, leaving a collection of 240 high-resolution photographs (2304 × 1728 pixels) of entecavir crystals. Each crystal in these images is meticulously labeled with dots and lines, following the annotation format of the COCO dataset, with the details recorded in .json files. The labels distinguish between single-crystal and polycrystalline forms. A total of 180 images were designated for training and 40 for validation, and the dataset was randomly shuffled to ensure the robustness of the evaluation.

3.2. Experiment Equipment

RECDet is implemented using the Pytorch deep learning framework (version 1.7.0). The operating system is Windows 10. The CPU model is Intel Core i5 10th and the GPU model is Nvidia RTX3060. The images were captured using the OPLENIC microscope photography system (CX23) (OPLENIC, located in Hangzhou, China).

3.3. Experiment Result

To demonstrate the effectiveness of RECDet, four general object detection models are compared, including two-stage object detection [32,33], single-stage object detection [34], and Transformer-based object detection [35]. The codes to run these models are obtained from the OpenMMLab GitHub repository [36], the detailed list of hyperparameters is shown in

Table 2. Hyperparameters.

Parameters	Values
Optimizer	SGD
Learning rate	0.02→0.002
Weight decay	0.0001
Momentum	0.9
NMS	0.5

.

The accuracy of prediction boxes, category judgment precision, training duration, and prediction speed (Fps) are compared. Among them, bbox mAP75 and classification accuracy are commonly used as evaluation metrics. bbox mAP75 refers to the mAP value (Mean Average Precision) at an IoU threshold of 0.75. For each predicted bounding box, both precision and recall are computed. AP denotes the area under the Precision-Recall curve, and its representation is

$$\left\{\begin{array}{c} Precision={\displaystyle \frac{TP}{TP+FP}}\\ Recall={\displaystyle \frac{TP}{TP+FN}}\end{array}\right.$$

(12)

Considering the binary classification task in our experiment, the representation for accuracy (acc) is detailed as

$$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(13)

where TP represents True Positives, TN denotes True Negatives, FP indicates False Positives, and FN means False Negatives. The detailed results of the comparison are presented in

Table 3. Comparison of experimental results with different methods.

Data	Method	bbox_mAp	bbox_mAp75	Acc (%)	Time/Epoch (s)	Fps
Entecavir pharmaceutical crystals	RECDet	0.723	0.762	98.2	31	18
	Mask RCNN	0.691	0.739	97.2	75	11
	Faster RCNN	0.684	0.731	97.0	40	14
	YOLOF	0.677	0.686	94.3	26	17
	DETR	0.667	0.699	96.3	80	8
Glutamic acid crystals	RECDet	0.632	0.651	97.5	39	18
	Mask RCNN	0.593	0.636	96.2	80	11
	Faster RCNN	0.606	0.615	96.3	51	13
	YOLOF	0.583	0.582	93.2	30	16
	DETR	0.586	0.611	95.9	96	8

.

It can be seen from Table 3, RECDet achieves an approximate 2% improvement over Mask R-CNN in the bbox mAP75 metric. Additionally, it not only improves the prediction speed but also boosts the prediction accuracy by 1%, achieving 98%, which also surpasses other comparison models. Such enhancements are crucial in applications with high-precision requirements [37]. Moreover, these enhancements can yield substantial benefits when handling large volumes of data, which can reduce the overall false positive rate and improve computational efficiency.

For the accuracy comparison shown in Figure 6, RECDet achieves a faster training speed and higher training stability as well as better prediction accuracy. This is because the features of the crystal are utilized to preprocess the image content, which enhances the neural network’s learning effect on crystal images, ensuring better accuracy in the early stages of training. When these processed crystal image features participate in the neural network training process, they yield a stable learning curve, minimize the background noise, and lead to fast learning and enhanced accuracy. The acc, bbox mAP75, and Fps comparisons by our RECDet and other models are shown in Figure 7.

In Figure 8, it is evident that nearly all the crystals have been clearly annotated, with the details of images further delineated in subsequent figures.

In Figure 9, a magnified view of image details is presented. It is evident that the prediction boxes are accurately annotated, and the crystal types are identified correctly. Figure 10 displays the detection results in regions with dense crystal formations, which also exhibit a satisfactory performance of our RECDet.

3.4. Ablation Experiment

Ablation experiments evaluate how the binary maps for distinguishing background from the foreground and especially the designed detection boxes impact the model performance. The effects of removing the specially designed detection box (No box), removing the binary map to aid training (No bin), and the experimental effect of simultaneous removal (Null) are compared in the ablation experiment. As shown in

Table 4. Comparative results of ablation experiments.

Data	Method	bbox_mAp75	Acc (%)
Entecavir pharmaceutical crystals	RECDet	0.762	98.2
	No box	0.749	98.0
	No bin	0.757	97.3
	Null	0.731	97.1
Glutamic acid crystals	RECDet	0.651	97.1
	No box	0.629	96.6
	No bin	0.646	96.4
	Null	0.615	96.3

, the observed decline in accuracy across all scenarios underscores the effectiveness of our RECDet.

3.5. Experimental Discussion

The biggest advantage of the RECDet model is that it eliminates background interference and saves computational resources. Traditional object detection models require resource allocation for foreground and background regions. The RECDet model intentionally avoids detecting background regions, which can significantly reduce the computational complexity and increase the prediction efficiency. As shown in Table 3, RECDet nearly doubles the prediction speed compared to other models. Besides, background interferences often introduce noise into the detection process, leading to false positives or missed detections. RECDet ignores these background regions which can minimize the interferences. In addition, the specially-designed detection box designed by RECDet further improves the prediction accuracy. These superiorities of RECDet in crystal object detection are well demonstrated according to the above experiments.

4. Conclusions

This paper introduces a novel method for crystal object detection. It utilizes a two-stage detection process combined with the Otsu threshold segmentation algorithm to adaptively distinguish between background and foreground regions in the crystal image, which helps our neural network to accurately predict the crystal shape and reduce the interference from impurities and background light intensity. Additionally, the background region is omitted during the candidate box generation process, minimizing the use of computational resources. Furthermore, the specially designed candidate box is well-suited to the crystal size, allowing for faster adaptation by the model. Compared to the current Mask RCNN model, the accuracy of the candidate box region is improved by more than 2%, and our classification accuracy reaches 98%. Finally, a self-made entecavir pharmaceutical crystals dataset is constructed for use in crystal object detection. The information including RECDet and the crystal image dataset is available at https://github.com/Alxcial/RECDet (accessed on 30 October 2023). The dataset will be made available for public use.

Certainly, there remains room for enhancement. The application of artificial intelligence technology in the actual crystalline production process still faces many challenges. These issues include blurred photos due to environmenal interference, uneven brightness caused by water ripples, problems related to turbid solutions and crystal agglomeration, and challenges associated with insufficient data. In our future work, we will continue to refine the method proposed in this study to solve the problems encountered in real production environments and mitigate the effects of these external factor disturbances.